Random thoughts in no particular order:
It seems like your roommates have two different concerns -
1. As your friend, they may have concerns of possible sexual addiction or other problems manifesting itself in outlandish behaviors.
2. They feel that X number of people live in the house, and that by having a constant parade of new people, the house actually contains X+Y people where Y is always changing yet ever present. Y probably does not contribute to rent or chores, but no doubt contributes to the scuff marks on the floor, the reduced toilet paper volume, and may take up prime couch space. This may lead to resentment of Y that is then transferred to you.
3. The difference between having a friend over for dinner and a trick is that with the former, housemates are generally welcome to participate in the social aspects of the evening (anyone want to play cards?) wherein in the latter, housemates are generally expected to be quasi-invisible (lest your trick be distracted engaging in conversation with them instead of groping you).
4. The discomfort that you feel at going to a stranger's house to trick is likely similar to the discomfort your roomies feel when there is some random guy on their couch.
Just my thoughts.