While there is an enormous amount of possible passwords, there is only a limited (and quite small) amount of users. Couldn’t you just hash all the usernames one by one and map the hashes to the usernames? So you could still reverse engineer the usernames of those who voted on a post.
Edit: Salting with the post id would make this attacking process harder, but still realistic. Probably the only real solution is to hide the votes table from federated instances, I’m not sure if that brings technical problems.