Not likely worth it. Primary reason is that the large federated email services are skeptical of email from services such as your proposed self hosting solution and may simply not deliver the mail you send. This is to mitigate against spammers setting up bespoke servers.
There are a bunch of other things that could go wrong if you don’t set everything up perfectly, but even if you do, this would be a big problem.
Better off using a custom domain with a big provider. Fewer headaches. I like Fastmail, but many others are great too.