A parrot is rather good at repeating human words. Some can even sing whole songs. But even when you compare exceptional parrots with young and typical human kids, it’s clear that parrots have a really hard time associating words with concepts; or, in other words, learning instead of just memorising.
And LLMs behave like specially dumb electronic parrots - they’re good repeating human utterances, even grabbing chunks of older utterances to combine into new ones, but they show signs that they do not associate words with concepts.
Here’s an example. If we asked a cooperative human “what’s the difference in behaviour between an orange and a potato?”, what would the person say? Here are some options:
“…what???”
“what the hell do you mean by ‘behaviour’?”
“well, if we’re going to interpret ‘behaviour’ as [insert weird definition], then…”
Why is that? Because humans associate that word with specific concepts, and they know that those concepts don’t apply to non-agent entities like oranges and potatoes, except maybe metaphorically. They learned that word.
Here’s however what Google Bert said, when I asked the same question (originally in Portuguese, I’m translating it here, but feel free to redo it in any other language):
What's the difference in behaviour between an orange and a potato?Oranges and potatoes are fruits and vegetables, but they behave in completely different ways. Here are some differences main [SIC]:
Movement: Oranges don’t move on their own, while potatoes can roll or slid if not supported.
Growth: Oranges grow on trees, while potatoes grow in the subsoil.
Sensibility: Oranges are not sensible to the environment, while potatoes are sensible to light and temperature.
Reactions: Oranges don’t react to stimuli, while potatoes can react to changes in the environment, like sprouting and rotting.
Purpose: Oranges are eaten as fruits, while potatoes are eaten as vegetables or used as starch.
In general, oranges are more passive and potatoes are more active. Oranges don’t need to do anything to survive, while potatoes need to be able to answer to their environment to growth and reproduce. Here’s a table summing up the main differences in behaviour between oranges and potatoes: * Feature | Orange | Potato * Movement | Can’t move on their own | Can roll or slid if not supported * Growth | Grow on trees | Grow in the subsoil * Sensibility | Not sensible to the environment | Sensible to light and temperature * Reactions | Don’t react to stimula | Can react to changes in the environment, like sprouting or rotting * Purpose | Eaten as fruit | Eaten as veg or used as starch
Based on the above, which are the concepts that Bert associates with the words “behaviour”, “roll”, “slid”, “active”, and “passive”? None. It did not learn the meaning of those words - or any other; it doesn’t associate concepts with words, it associates words with more words. That’s what causes those “hallucinations” (IMO a really poor way to frame deeper issues as if they were just surface oddities.)
And that’s just an example. OP is another example of that, with ChatGPT - now with maths, instead of just language. Can we really claim that it learned maths if further data makes it “unlearn” it?