It’s kinda worse than that — it’s used to authenticate yourself as a citizen.
My SSN should at most be an ID, no different from a name. I can identify myself as Darth Vader or 4200-69-1337, but that shouldn’t matter, because I should never be able to authenticate myself as either of those.