I get what you’re saying. There’s a mindset in the fediverse that everyone on an instance is responsible for it. Even if the bad actors join later. The instance gets defederated but as a user it can be really hard to know if your instance is defederated.
It’s a deeply unpopular opinion and anyone who suggests that federation is simultaneously a huge advance and a big problem seems to get downvoted.
You’re right, ultimately instead of being exposed to a range of views some of which are challenging we’ll end up in little echo chambers.