They can have whatever they want, but you’ll have to forgive people for thinking that you align with people who display the same symbols as you.
I assume anyone flying a swastika is antisemitic, when to be fair, they might just be a fan of the Nazi stance on affordable housing and infrastructure.
If you have a problem with symbols you identify with being co-opted by people you don’t, take it up with the people you disagree with who took your symbol, not the people who also disagree with them.