If you saw a house burning down and you did nothing to stop the flames, wouldn’t you say that that person wanted the house to burn down?
If you are not actively against white supremacists, you are allowing them to spread and flourish. It’s like a weed. If you don’t pull it, it’s going to take over your beds.