The problem is that everyone has consolidated on one gargantuan server. The whole point of the fediverse is to spread out so no one server is carrying the entire load. I’m currently using lemm.ee and have experienced none of the issues being discussed here.
But yes, I agree that it could be a potential turn off for newcomers.