To be overly simple about it, the LLM uses statistics and a bit of controlled RNG to pick its words. Words in the LLM have links to each other with statistical probabilities attached. If you take the sentence “I fed a peanut to an elephant” and “I fed a peanut to a elephant” and then asked 100 people which is more correct, there will be a percentage which favors one over the other. Now with LLMs its not choosing using weighted coin flips, but rather picking the most likely next word (most of the time). So if the 100 people choose “an elephant” over “a elephant” 65% of the time in its training data, then the LLM will be inclined to use “an elephant.” However, Its important to know that the words around “an elephant” will also bias its choice to use the word ‘an’ for the word ‘elephant’.
Really, its largely based on the training data and the contexts to which ‘a’ and ‘an’ are used. Or in other words, the LLM knows because people figured it out for the LLM. People did all the thinking, LLM’s just use statistics on our bottled phrases to know when to use which. Of course, because it got its data from people - it will sometimes get it wrong which is based on how often people got it wrong.