Youre right, we do just use language to describe things in a convenient manner that is not actually universally true. Do you think language just springs out of the ground or something? Humans make it. We make it socially. One might say we socially construct these concepts.
Biological sex is not a thing. There are people with dicks and people with vaginas and people with neither and people with both and people with stuff that isn’t even classifiable in terms of the terms dick and vagina. Why is there not a sex for each possible genital configuration? Why not one for each possible chromosomal configuration? Because sex is a concept we as humans created that does not map 1 to 1 with biological reality. Biological sex is not a thing, there is biology and then there is the human made concept of sex. They are 2 different things.
Your doctor assigns you a gender at birth. In most countries he is legally required to mark down one at the time of your birth. That gender is used for all the reasons I listed in a previous comment. Your mother then picks you up and affirms that gender assessment. From then on your gender is assigned until you yourself revoke it.