Because any good people in the group would’ve noticed the swastika flags and decided to do something else that evening or ideally joined the counter protests.
I’ve been a counterprotestor before, you generally know who’s on your side and what it looks like on your walk there. You have time to see who’s with you.
It wasn’t like it was a handful of Nazis and a lot of people who just wanted lower taxes. It was a night that culminated in a large mass of men chanting that Jews will not replace them as they carried torches and many carried swastika flags. There may have been bad people on both sides, but only one side had good people, the other had too many Nazis to have any good people.