Here’s an example because changing context might help:
I’m a white guy
But I’m not a white supremacist.
That doesn’t stop me from being in their “core group” which is white people.
That doesn’t mean every white person is a white supremacist. But if a white person speaks out against white supremacy, they’ll hate you even though you’re white.
Often they take that really hard. The KKK killed a lot of white people, in some areas all they killed was white people who disagreed with them, because no Black people could live there in the first place.
Its why people always need to speak out against that shit, even if you’re not in a targeted group, if they ever cross your personal moral line and you speak out, you’re immediately one of “them” and not “us”.
And fascists always need a “them” so they’ll keep moving people out of “us”.