With the current way that ActivityPub works, this isn’t really possible. Every vote needs to be signed by some real user; if that changed such that anonymous votes were accepted then there’s nothing to stop any random person from adding 5 or 5,000 anonymous votes.