Right. It happens to symbols that aren’t widely used. Fascists couldn’t co-opt the Disney logo or something. Even if copyright or trademark wasn’t a thing, no one would cede it.
The most common thing I can think of is the OK symbol, which people mostly didn’t let them take for very long.