This is pretty surprising to me. In my experience (as a woman myself) women are much more likely than men to be vocally supportive of treating sex work like any other service and of breaking the taboo of offering or receiving those services.
I actually can’t think of any woman in my life who would judge someone negatively for seeing a sex worker (assuming full consent from all involved parties including partners). Most men I know would similarly have no issue with it, but a handful would read it as not being able to get laid and see that as something negative.
My social circle isn’t representative of the general population, but I’m still surprised to hear your experience is dramatically different. I wonder if the way the conversations are going make the issue more about consent, cheating, or other non-sex-work-specific ethical questions.