Admins can and do ban accounts that downvote rampantly
Obvious bot brigading is obvious. It became harder to tell on reddit when they started fuzzing the vote numbers, but could frequently still be figured out. It’s easier on Lemmy, someone just has to report some unusual voting pattern to the admin and they can check if the voting accounts look like bots.
I was once told that the algorithm is less weighted towards upvoted comments and more weighted towards recent comments on Lemmy, when compared with reddit. I am not sure if this is true, but I have noticed that recent comments tend to rise above the top upvoted comments in threads when viewing by Hot.
Without any way for bad content to be filtered out, you just end up with an endless stream of undifferentiated noise. The voting system actually protects the platform from the encroachment of bots and the ignorant mob, because it helps filter them out from the users who have something of value that they want to contribute.