AFAIK, there is no current recourse except defederation and defederation would be very slow and depend on every individual instance defederating. As well, there’s plenty of instances that haven’t defederated from the literal nazi instance, so who’s to say that they’d defederate from a bot heavy instance, either? Especially if the spammer would to invest even the slightest effort in appearing like there’s at least some legitimate users or a “friendly” admin. And even when defederation is fast, spammers could turn up an instance in mere minutes. It’s a big issue with the federation model.
Let’s contrast with email, since email is a popular example people use for how federation works. Unlike Lemmy (at least AFAIK), all major email providers have strict automated spam filtering that is extremely skeptical of unfamiliar domains. Those filters are basically what keep email usable. I think we’re gonna have to develop aggressive spam filters soon enough. Spam filters will also help with spammers that create accounts on trusted domains (since that’s always possible – there’s no perfect way to stop them).
I’m of the opinion that decentralization does not require us to allow just anyone to join by default (or at least to interact with by default). We could maintain decentralized lists of trustworthy servers (or inversely, lists of servers to defederate with). A simple way to do so is to just start with a handful of popular, well run instances and consider them trustworthy. Then they can vouch for any other instances being trustworthy and if people agree, the instance is considered trustworthy. It would eventually build up a network of trusted instances. It’s still decentralized. Sure, it’s not as open as before, but what good is being open if bots and trolls can ruin things for good as soon as someone wants to badly enough?