The AI guessing portion is currently playing on ultra-easy mode because there are only a few useful LLMs to compare to, and none of those have given serious thought to strongly securing their tech demo solution communications channel yet.
This is important work though, since someday someone might use an AI for something important and actually want to prevent eavesdropping.
I’m being a little unfair, but it’s early days and let’s try to not take ourselves too seriously just yet. If I’m trusting one of these LLMs with something deeply sensitive, it’s not eavesdropping that’s going to get me into deep shit, it’s the LLM’s hallucinations.
This is important research, though. Someday the AI will get good, and I’ll want to chat with it securely.
Thankfully, I suspect mitigation will be quite straightforward. Carefully designed small random changes to token handling should throw this approach way off, while not even being noticed by end users.
The habit of sending tokens right as they generate is a dumb sales gimmick that needs to go away anyway. So if security folks manage to kill it, good riddance.