I think your idea is not necessarily wrong but it would be hard to get right, especially without making the entry into fediverse too painful for new (non-tech) people, I think that is still the number one pain point.
I have been thinking about moderation and spammers on fediverse lately too, these are some rough ideas I had:
Ability to set stricter/different rate-limits for new accounts - users older less then X can do only A actions per N seconds [1] (with better explained rate-limit message on the frontend side)
Some ability to not “fully” federate with too fresh instances (as a solution to note [1])
Abuse reputation from modlog/modlog sharing/modlog distribution (not really federation) - this one is tricky, the theory is that if you get many moderation actions taken against you your “goodwill reputation” lowers (nothing to do with upvotes) and some instances could preemptively ban you/take mod action, either through automated means or (better) the mods of other instances would have some kind of (easy) access to this information so that they can employ it in their decision.
This has mostly nothing to do with bot spammers but instead with recurring problem makers/bad faith users etc.
Though this whole thing would require some kinds of trust chains between instances, not easy development-wise (this whole idea could range from built-in algorithms taking in information like instance age, user count, user age and so on, to some kind of manual instance trust grading by admins).
~
All this together, I wouldn’t be surprised if, in the future, there will eventually be some kinds of strata of instances, the free wild west with federate-to-any and the more closed in bubbles of instances (requiring some kind of entry process for other new instances).
[1] This does not solve the other problem with federation currently being block-list based instead of allow-list based (for good reasons).
One could write a few scripts/programs to simulate a federating instance and have tons of bots ready to go. While this exact scenario is probably not usual because most instances will defed. the domain the moment they detect bigger amount of spam, it could still be dangerous for the stability of servers - though I couldn’t confirm if the lemmy federation api has any kind of limits, can’t really imagine how that would be implemented if the federation traffic spikes a lot.
(Also in theory one could have a shit-ton of domains and subdomains prepared and just send tons spam from these ? Unless there are some limits already, afaik the only way to protect from this would be to switch to allow-list based federation.)
Lot of assumptions here so tell me if I am wrong!
Edit: Also sorry for kind of piggy-backing on your post OP, wanted to get this ideas out here finally