functionally the downvote button is the “I had a negative reaction to this button” and the up vote is the “I had a positive reaction to this button”
Exactly this.
Some of my favorite comments have loads of downvotes and to me it just makes whatever dumb shit I said funnier
If you’re talking about your comments, and you laugh at others who downvote them, then well done. But I submit that most humans are not so thick-skinned.
There’s also the question of visibility: downvoting a comment is not just saying “You’re wrong”, it’s making the comment less visible to others. A form of censorship, basically. I personally find it extremely annoying and unfair when it happens.
Perhaps it also has to do with how much time and thought you put into the offending comment.