This would solve some of the problems. If only 2 instances know about the votes, post instance and sublemmy instance, you can reasonably expect to get most instances to never release that info. It would allow either the sublemmy or post instance to manipulate around in the votes, but most manipulation would be detectable by the respective other instance.
It would open the door however to manipulating around with internal posts made from the instance in a sublemmy on the instance. And it would allow the post instance to drop votes selectively, though I think that is possible currently all the same.
Votes being sent to both the sublemmy and the post instance simultaneously would make manipulation a lot harder. And for cases like internal posts, you could add another involved “judge instance” that receives the vote details directly from source, and is merely there to confirm the total. Instances that hand out non-independent “judge instances” could be labeled as untrustworthy in the lemmy community.
So you end up with a list of instances per post that votes are reported to, to which you add the post instance, sublemmy instance, judge instance, and maybe some more.
In terms of implementation, I think the activitypub protocol needs an origin for votes, right? I would say an instance can just report the votes coming from a stock of obviously fake accounts, like “masked_upvote_1” to _999999 … and “masked_downvote_1” to _XYZ.
About the votes, I am not sure. It could be done as a lemmy-internal feature where lemmy instances and other instances knowing of the lemmy protocol send the info to all the relevant instances, while any votes from external instances only arrive at I guess the post instance and that then forwards it on to all other instances. This way the checking doesn’t work for software unaware of that lemmy specific vote implementation, but everything is still compatible.
You could then even for those lemmy-external votes add an interface on the judge instance, that would confirm via pm if your vote has arrived.