It’s like the difference between “my shit”, “your shit”, and “that shit”. You’re not actually referring to your own things as feces, or calling it “shitty”. It’s just your shit. As in “Don’t touch my shit”. But when you’re referring to someone else’s shit as “your shit” or “that shit” it’s more derogatory. Like, “clean up that shit” or “get your shit out of here”.
The context changes “shit” from derogatory to neutral. Similarly, “dude” can be both gender specific and neutral depending on context.
Note that people are still allowed to prefer not to be referred to as “dude”, but it’s a gender neutral term in many contexts nonetheless.