It would have hurt early on, but no so much now. They locked the instance down, quit accepting new users, and banned users being able to create communities themselves stating that the goal of the instance was to be a carefully curated one and they considered it “complete” and no new communities would be created.
They’re not even in the top 15 instances by users anymore.
The biggest impact would be that Lemmy.ml is the developers instance but I imagine they have backups of everything and would just register on a different domain and transfer everything and everyone.