Unfortunately I think people downvoting things they disagree with is kind of inevitable. People are notoriously combative online, and if they’re given an option to drown someone out, they’re going to abuse it. And that makes it even easier for any sort of hivemind to kick in.
I personally don’t know a better system, but it’s not perfect.