I totally agree that LLMs don’t currently have a ‘concept’ of what they’re being asked. But it’s also pretty easy to ask questions of young kids that go beyond what they have a mental map of and get nonsense.
Here’s what ChatGPT has to say in response to your question:
An orange is a citrus fruit, typically sweet and juicy, while a potato is a starchy vegetable. They have different flavors, textures, and nutritional profiles. Oranges are typically eaten raw, while potatoes are commonly cooked before consumption, such as boiled, baked, or fried.
Me: But what about their behaviour?
I apologize for the confusion in my previous response. Fruits and vegetables do not exhibit behavior in the same way animals do. Behavior is a characteristic of living organisms that involves actions and responses to stimuli. Fruits and vegetables are plant-based and do not display behavior as animals do. They follow natural processes of growth, ripening, and decay but do not behave in the same sense as living creatures.
So, a smarter parrot? How far until it’s as good as any young kid? For the record, I’ve worked in this field for a long time and am far from claiming any kind of strong AI is even close. The tests the OP did are fragile by design so their result isn’t surprising.
And you still didn’t say anything about not using the data ‘well’. What would you like to see them doing?