If it was the pi 4, I would agree, but the Pi 5 would probably handle less than 20 people. I have a 4 that uses Yunohost that covers a LOT of services. I played around with matrix but it required too much out of the system. But that’s with a lot of other services running. Give it a shot and let the rest of us know.
BTW I had a lemmy/irc/bookwyrm/mastodon all running on the pi4 without any real issues. Mastodon is probably the beefiest/slowest app, and its been getting better as time goes on. So a PI can definitely work with fediverse apps.