I’d go the other way: make these things officially public, so people know they are, and then aren’t taken by surprise.
Private voting can be tricky in a federated setting, because I could have a malicious instance that boosts my posts (I can have it with public votes too, but then it’s easier to detect). Truly private posting history is outright impossible, as you said, due to crawlers.
The way to privacy is to make sure not to dox your account, perhaps alternate 2-3 accounts if it’s really important to you.