If you can’t trust peer review from experts in a field, many aspects of society break down. For example:
How can we trust the word of an engineer that says a bridge is safe? Did you verify the calculations yourself? Have you personally tested the tensile strength of that rebar? Better to just avoid bridges to be safe.
How can we trust the word of a doctor when they prescribe something? Did you personally look up all the possible side effects and made sure you’ll be safe? Do you research clinical trials yourself to verify efficacy? If you don’t trust your doctor, you’ll be right at home with the anti-vaxxers.
How can you trust a lawyer to argue your best case? There’s thousands of pages of law that most people haven’t read. Do you know for yourself that there isn’t some past precedent that completely flips your case? Defending yourself is a bad idea for a lot of reasons.
Nobody can be an expert in every field. It’s completely unfeasible for most people to verify source code themselves, but that doesn’t mean open source doesn’t matter. Society operates on a degree of trust in our fellow humans that ARE experts in their field. The more experts in agreement the better, since nobody is infallible.
I’m not sure what you’re suggesting people do? Go live in a hole by themselves because the world is full of liars and deceivers? Or become superhuman and hand verify every possible thing that could negatively effect them?