You can prove some things are correct, like math problems (assuming the axioms they are based on are also correct).
You can’t prove that things like events having happened are correct. That’s even a philosophical issue with human memory. We can’t prove anything in the past actually happened. We can hope that our memory of events is accurate and reliable and work from there, but it can’t actually be proven. In theory everything before could have just been implanted into our minds. This is incredibly unlikely (as well as not useful at best), but it can’t be ruled out.
If we could prove events in the past are true we wouldn’t have so many pseudo-historians making up crazy things about the pyramids, or whatever else. We can collect evidence and make inferences, but we can’t prove it because it is no longer happening. There’s a chance that we miss something or some information can’t be recovered.
LLMs are algorithms that use large amounts of data to identify correlations. You can tune them to give more unique answers or more consistent answers (and other conditions) but they aren’t intelligent. They are, at best, correlation finders. If you give it bad data (internet conversations) or incomplete data then it at best will (usually confidently) give back bad information. People who don’t understand how they work assume they’re actually intelligent and can do more than this. This is dangerous and should be dispelled quickly, or they believe any garbage it spits out, like the example from this post.