I think we are somewhat on the same page here. What matters isn’t the word you use, it’s the intention behind, which is hopefully obvious from context.
When I get called a nerd, my response goes anywhere between being proud and being insulted depending on context. The word itself has no emotion by itself.
Shouldn’t it be the same with many of these words that some people consider racist?