It’s true that we can always choose to defederate from them. What’s to worry about is their meddling with the ActivityPub standard using their incomparably vast resources, and them making their own extensions to the standard in efforts to suck users back into the Borg. Things like that.
Unfortunately, even if instance admins were to unanimously defederate, Meta—or any social media corporation—could create white-label instances to take their place, and we might be none the wiser of their control of them.