You could switch some of the problems with perf in switching away from the Python implementation server as well as Element clients but these support the most up-to-date features & the majority of users are now relying on these features that often don’t degrade graacefully.
The bigger issue is eventual consistency. Eventual consistency will not scale for small self-hosting. Every message & every attachment for every user in every chatroom they have joined must be duplicated to your server. This is why joining rooms sometmies takes 10 minutes. Even if you make this async from the client side instead of the current long wait, your server & storage are still taking the hit. A lot of small collectives had to drop their servers for performance & cost (read about yet another one today on the Techlore thread at c/privacy where now only Discord is used for realtime coms). This model is required to copycat the ability to search the entire history like the big, proprietary chat apps such as Slack/Telegram/Discord, but they are centralized so it is easier to manage—but its overuse for all announcement & trying to replace forums turns it into a black hole for information. Your small community probably does not need persistent chat like this—persistent info is lighter & easier to crawl as feeds & forums. With medium-sized servers shutting down, only the biggest & smallest hosts are still kicking with most metadata is largely centralized around Matrix.org who also hosts some of the other larger instances.
If you agree that chat can be chatter as well as ephemeral there is lightweight centralized chat in IRCv3 with TLS has most of the features you need with a longer legacy & massive choice for clients & XMPP for lightweight decentralized chat with a long legacy, client options too, & can be self-hosted in a bedroom on a toaster in comparison which increases the chances of self-hosters & decentralization. These were built in a time when we didn’t have such wasteful taste in tech since they needed to be efficient & only sip power/data in comparison both for clients & servers & storage. The bigger question IMO is what are fundamentally wrong with these two mature options that we need a new option built on unextensible JSON & Israeli Intelligence money?