Defederating from extremistic or spammy instances should not be hurting regular users.
What I find annoying is when defederation happens because of individual communities on otherwise good instances. E.g. when that drama about shitjustworks started, because somebody created an empty The_Donald community there…