I’m really skeptical about that. Either that they would do it or that such “justified” downvoting would be a clear cut or fair decision. Most people don’t vote the right way. How many people downvote content they agree with or find funny but doesn’t add to the discussion? How many people upvote content they disagree with that does add to the discussion?
And am I really going to take up a mod’s time because someone got mad at me and downvoted—the most accessible and innocuous was to express displeasure with someone? How many more complaints about downvote bullying are mods going to have to field?
I don’t know. You could be right, but I’d want to see it successful in a small scale, if possible, before deploying it everywhere. Maybe the folks suggesting it should be up to the server admin are right. That would be another differentiator and people could go to communities on servers that have their preferred visibility policy. That would serve as an A/B test and let people vote with their feet.