You’re completely correct. Normally, I’m on the side of not assuming people’s gender and I’m of the mind that you shouldn’t judge a book by it’s cover.
But, because of the safety and personal ramifications crossing the wrong person can have, I think it’s important that we acknowledge a woman’s right to seek safety in a situation she perceives might possibly go south for her. That includes the prerogative of not putting herself in a situation that she perceives as risky to begin with.
Maybe that concept would be better accepted if it were expanded to “Anyone should have the right to avoid danger they think they might be in”