Yes, some people do misuse down voting, but it’s absolutely worth it in my mind. It prevents bigotry from both being shown highly and also makes it clear that bigotry isn’t acceptable. How do you think someone feels when they see a bigoted comment and it’s in the positives. I’ll answer that question cause I’ve seen it all the time: it’s depressing as fuck. It still sucks to see bigoted comments at all, but at least when they’re downvoted to hell, it feels better. It feels nice to know that the bigotry isn’t accepted by others and that the bigot is in the minority.
Similar for general misinformation, scams, etc. It’d be nice if bad comments could just get removed before you see them, but that will never be perfect. Even well moderated communities still have delays before moderators see reports and most people won’t take the time to report either.