At the moment, upvotes and downvotes, while not used that way by many people, is more about what others will see, rather than what content you like. It’s more like a community moderating and rating effort. Upvotes make posts more visible, by pushing them further up in what’s currently popular. Downvotes do the opposite, and in my personal opinion, should be reserved for posts that don’t fit the community they were posted in, spam, or things that break rules – typically the same reason why you would (and should) report a post. They are not “agree” and “disagree” buttons. Topics you disagree with can still spark interesting conversations.
Using the same mechanic, voting, to tell an algorithm whether similar posts should have higher visibility on your own feed, would be incompatible with this existing system. Posts that get a quick reaction or emotion out of you are even further encouraged, while things you simply don’t want to see (but aren’t necessarily “bad”) get punished heavily.
This system works through subscribing to communities you are interested in and actively participating in improving the health of those communities, rather than passively consuming content. That takes some effort, yes.
All in all I think this proposed system is not compatible with Lemmy, and maybe not even a good idea.