Your question would be much better applied to height discrimination, which is something that’s almost never mentioned, but is a lot more indicative of the nature of discrimination itself.
It is instinctual, as others have said, but it has nothing to do with tribalism or war, its about resources. Discrimination is almost always about resources (the notable exception being gender/orientation based discrimination, which I guess is religious?).
The discrimination against small people (and obesity and age as well), is more basic, and likely older (in evolutionary terms), and is oriented towards hunting and fighting. We think less of smaller, fatter, and older people because they’re assumed to be less capable of gathering (and fighting for or defending) basic resources.
Discrimination against races is more recent, and more societal, and is more about monetary resources, and isn’t even entirely a matter of race. Poor white people can be discriminated against in the exact same way for the exact same reasons. Racism is more classist than discrimination against height, weight age, etc. but is essentially still a matter of these classes being seen as less capable of getting resources.
You can see it more easily if you look objectively at the discriminatory tendencies of women (and I mean that in a very generalized way). They tend to be far more discriminatory towards resource based biases… Height, weight, physical condition… They’re often inexplicably attracted to overly aggressive partners, occasionally to their own detriment. The more instinctual a woman is, the more likely to pursue the overly aggressive men. Race isn’t anywhere near as much a factor, and there are notable exceptions in all factors for women if a man obviously has a lot of resources already (no indictment intended ladies, just is what it is, and generally)
And of course it’s more obvious among women for the same reason… The disparity (again, in a very general sense) between male and female in ability to gather and defend resources affects women’s choices of partners more so than men.