Especially because you might not know why someone was downvoting the post/comment.
Case in point: lemmy.ml/comment/1009973A decent number of users thought it was a good idea to downvote this post because they didn’t want to give Nazis more publicity. However, someone could take a look at all of the downvotes on this post and make the assumption that everyone who downvotes it is a Nazi themselves.