I like the idea and how it’s presented. Yes, I think this might work.
Then I asked myself why we need it and if we really lack tools to solve it.
What I mean is: Instances can already close themselves for new registrations. Or they can make their signup process less easy (for example by requiring a written statement why the user wants to join specifically them).
If instances don’t do that, would they opt into your system instead? After all, both depend on the instance to want to grow slower. In both cases, I’d expect those instances to become the biggest who simply accept any new signup for themselves.