The article makes several claims and insinuations without backing them up so I find it hard to follow any of the reasoning.
I don’t think it’s desirable that it’s easier to reason about an AI than about a human. If it is, then we haven’t achieved human-level intelligence. I posit that human intelligence can be reasoned about given enough understanding but we’re not there yet, and until we are we shouldn’t expect to be able to reason about AI either. If we could, it’s just a sign that the AI is not advanced enough to fulfill its purpose.
Postel’s law IMHO is a big mistake - it’s what gave us Internet Explorer and arbitrary unpredictable interpretation of HTML, leading to decades of browser incompatibility problems. But the law is not even applicable here. Unlike the Internet, we want the AI to appear to think for itself rather than being predictable.
“Today’s highly-hyped generative AI systems (most famously OpenAI) are designed to generate bullshit by design.” Uh no? They’re designed with the goal to generate useful content. The bullshit is just an unfortunate side effect because today’s AI algorithms have not evolved very far yet.
If I had to summarize this article in one word, that would be it: bullshit.