I think the advice should be taken to heart here — you’re dealing with a userspace problem but you’re trying to get the kernel to make it all better.
You’ve already mentioned the two big things, compressed RAM and swap; optimizing userspace (or paying for more RAM) may be the only option at some point.
If you want to get creative, is there a reason you can’t use a local computer for some of these services? An old raspberry pi or similar could potentially run some of your services. You could run some containers on your home server and call it a day. Quick search turned up this linuxserver.io/…/routing-docker-host-and-containe…