You sound odd, like a child or someone not fluent if you don’t use our misuse the genders of words.
That being said, as native Spanish that lived in the UK for a while, I noticed that genders and verb forms are useful for providing more context when talking.
Cannot think of specific examples now, but in general in a phrase if you don’t hear a word or don’t know the meaning, it is easier to guess it because the rest of the phrase is constructed around the gender and more complex verbal forms.