It affects those who don't want their votes visible and for random stranger to track their activities. That's why this is better as an opt in for those who want them and have them be invisible for those who don't want to share it. Think of how you can set your Youtube playlists like favourites to private or unlisted.