Freedom of speech almost always comes with freedom of association. The thing that many people miss is that freedom of association also allows you to choose NOT to associate with someone just as freedom of religion includes freedom FROM religion. The instances choosing to defederated from the trolls are exercising their right to be free from association with the trolls. No one is stopping the trolls from spewing their nonsense or stopping anyone who wants to listen from listening.