“karma” (as reddit calls scoring) never was more true to its name. :)
I haven’t looked at Lemmy’s implementation of upvotes/downvotes, but they should be ActivityPub activities, so it means they should appear by making a request to the user’s actor.
EDIT: I’ve just checked random users outbox (that’s the ActivityPub name for the list of activities), included mine, they are actually just empty. So that probably means that Lemmy is only publishing the upvotes/downvotes when pushing activities to federated servers, which would make those activities way more private, although not completely : someone could setup their own instance to learn about them, and it’s best to be assume that at some point, someone will start such instance and publish an app revealing all votes for everybody (plus, as others mentioned, Kbin is already doing it).