I have a counter example where downvotes are very useful. In r/askphysics we would downvote any comment that had errors or was completely wrong. A post didnt have to be around long before it became clear what the consensus answer was. This was much more helpful to the asker than just showing them a bunch of answers, and responses to those answers, and leaving them to discern who is most credible.
In that case, just not voting wouldn’t help. And if you have downvoting, you can still just not vote when that’s more appropriate. That’s what I do for opinions I simply disagree with.
ShipHeadlights