I basically agree with you except for one caveat. I would call the “grouping” that you describe as gender expression. Whether men where kilts or pants is mostly based on societal expectation. But the unchanging gender identity is not a social construct (in my view).
That said, I raise the question because I am open to having my views changed. They’ve certainly changed in the past.