Do you not know how often racists use crime stats to bolster their arguments too?
Men more often kill women is no grounds to make blanket arguments about men in general, same way as some minorities being overly represented in rape statistics is no grounds to make blanket statements about those minorities.
You can say it’s just you “being careful” and whatnot, but you should realize that you’re using word for word the same arguments racists use to justify their racism. Both represent an actual threat to women for sure but it’s no justification for labeling them all, that’s the point.