I think there should be an option (unless there is) for mods to turn off (or hide) voting as needed. That might be an effective way to cancel any downvote brigades. Lemmy really doesn’t have the population for mass vote manipulation now, but it will soon enough.
Hiding all votes can also help mitigate some superficial bias, but not all. I believe that if a person sees a comment with a few dozen downvotes first, they tend not to read the post objectively. After being on Reddit for such a long time (12 years or so), I found that it was super easy to manipulate voting trends if I caught a post or comment at just the right time.
Hiding only downvotes is just silly though. Some register of public opinion, positive or negative, still has its uses, IMHO.