This is the same middlebrow dismissal that AI advocates have been using for years.
“It’s just a stochastic parrot.” “How do you know that you aren’t just a stochastic parrot?”
Well we do know. There are experts on human cognition. They have been studying it for decades. We may not know enough about it to know how to make a computer do it. But we certainly know enough about it to know when a computer chatbot is not doing it.