Yes and no. The people still have the voice, you’re not preventing them from talking to other people, you’re just preventing them from accessing your instance.
Centralized websites like Facebook, Twitter, whatever. That would be censorship if they remove somebody’s voice.
Decentralized instances, are closer to private communities. They’re focused, and they create curated viewpoints.
If the federated universe gets dominated by one or two powerful instances, then yes it would be censorship, but if people are using it at a decentralized fashion: it’s just communities expressing preferences.