They produce this kind of output because they break doen one mostly logical system (language) onto another (numbers). The irregularities language has get compensated by the vast number of sources.
We don't need to know more about anything. If I tell you "hey, don't think of an Apple", your brain will conceptualize an Apple and then go from there. LLMs don't know "concepts". They spit out numbers just as mindlessly as your Casio calculator watch.