It’s risky but the risk is towards the users, and the profits are towards the companies.
I asked OpenAI, Google, and Meta what they are doing to defend against prompt injection attacks and hallucinations. Meta did not reply in time for publication, and OpenAI did not comment on the record.
Discourse analysis tip: what is not said is sometimes more important than what is said. The fact that they refused to reply hints that the reply would be against their best interests, either lying in a liable way or saying the truth and potentially ruining their investment.
The reason why Google actually answered it (“Google confirmed it [prompt injection] is not a solved problem[…]”) is likely related to saying “it’s an experiment” -
Regarding AI’s propensity to make things up, a spokesperson for Google did say the company was releasing Bard as an “experiment,” and that it lets users fact-check Bard’s answers using Google Search. “If users see a hallucination or something that isn’t accurate, we encourage them to click the thumbs-down button and provide feedback. That’s one way Bard will learn and improve,” the spokesperson said.
Can we [people in general] stop pretending that those models “learn”? Giving it feedback is like telling my cat “don’t scratch it!” - it might work for that specific case, but it won’t solve the underlying issue, so the model/cat will keep hallucinating/scratching something else. The hallucinations are not individual flaws, they’re issues surfacing from the underlying tech: language associates morphemes (tokens) with meaning, not just a token with another! Linguists have been talking about this for at least a century, but those “tech bros” are still trying to model language without it. (Microsoft is apparently doing some progress in this regard though. I can look for the quote if anyone wants.)