The issue with reliability is a completely different one between web search and AI.
If you search something on Google, there are quite a few ways you can judge the quality of the answer with “metadata” around it. If you find a scientific paper, it’s probably more reliable than a post on a parents forum. If the source is a quality newspaper or Wikipedia, that’s also more on the reliable side, but some conspiracy theorist website is not. And if the source is some kind of forum or Q&A site, wrong answers often have comments under them that correct the error.
Also, you can follow multiple links and take a wider sample on the topic that way.
With AI that’s not possible. Whether it is wrong or correct, the AI will give you an answer in the exact same format, with the same self-confident tone. You basically need to know the correct answer to know whether the answer is correct.
Sure, you can re-roll and ask it again, but that doesn’t make the result more likely to be correct.
For example, I asked ChatGPT which Harry Potter chapter is the longest. It happily gave me a chapter, but it wasn’t the longest. So I asked again and again and again, and each time it gave me a new wrong answer, every time with made-up word counts.