I totally missed that you have an uncontainerized service. Can you run the service directly on the hardware host (safely)? If so, here’s how I would probably run it considering your memory constraints:
Install Debian
Set up the uncontainerized service directly on the host.
Install Docker directly on the host as well.
Spin up the containers for everything else.
Not the cleanest/most separated answer but it would reduce the memory load of additional layers of host/VM/containers. If this isn’t storing any sensitive data or being directly exposed to the internet that should be fine.
If you are dealing with sensitive data or exposing to the internet, I would consider your original plan of Proxmox VMs to separate everything but see if you can add additional RAM to help. Also consider installing something like fail2ban on every host and VM.