Many Lemmy instances do just fine without them though, and unpopular extremist views are still unpopular. Frankly that sounds more like a case for moderation than downvotes.
One of the main problems I have with downvotes on Lemmy is that when people browse All, niche communities tend to attract a lot of drive-by downvotes (which is why many instances that host them opted to disable downvotes) that tend to drown out votes by people who are actually in those communities and push the content lower when using a sort that takes votes into account.
Yes there’s all sorts of lofty ideas about how downvotes should be used, and eg people are not “supposed” to downvote things just because they disagree (and no I’m not talking nazism here). Never goes that way in real life.