I think what you’re looking to do is route using IPTables. I’ve achieved a similar setup with this guide, just not using a mail server. With this setup the DNS can actually be taken care of by docker. With my phone on wireguard I can resolve by the container name on my VPS, internal server docker container, internal lan, and everything else goes out to Mullvad (direct too thanks to split tunneling). Very slick setup.