Yeah I read one of the papers that talked about this. Essentially putting AGI data into a training set will pollute it, and cause it to just fall apart. Most LLMs especially are going to be a ton of fun as there were absolutely no rules about what to do, and bots and spammers immediately used it everywhere on the internet. And the only solution is to… write a model to detect it. Which then they’ll make models that bypass that, and there will just be no way to keep the dataset clean.
The hype of AI is warranted - but also way overblown. Hype from actual developers and seeing what it can do when it’s tasked with doing something appropriate? Blown away. Just honestly blown away. However hearing what businesses want to do with it, the crazy shit like “We’ll fire everyone and just let AI do it!” Impossible. At least with the current generation of models. Those people remind me of the crypto bros saying it’s going to revolutionize everything. It might, but you need to actually understand the tech and it’s limitations first.