If it doesn’t matter which bathroom people use, wouldn’t such a person use the women’s restroom if they were worried about that? It’s not like bathrooms are suddenly going to have an even mix of genders using them. I’m a guy. I’m going to use the men’s restroom either way. That’s going to be the same for almost everyone.
Also, if they want to fight someone for being trans, it’s going to happen regardless of the bathroom situation.