yogthos OP , 1 year ago

The fundamental problem is that at the end of the day it’s just a glorified Markov chain. LLM doesn’t have any actual understanding of what it produces in a human sense, it just knows that particular sets of tokens tend to go together in the data it’s been trained on. GPT mechanic could very well be a useful building block for making learning systems, but a lot more work will need to be done before they can actually be said to understand anything in a meaningful way.

I suspect that to make a real AI we have to embody it in either a robot or a virtual avatar where it would learn to interact with its environment the way a child does. The AI has to build an internal representation of the physical world and its rules. Then we can teach it language using this common context where it would associate words with its understanding of the world. This kind of a shared context is essential for having AI understand things the way we do.