My memory idles on around 3341MiB with a browser and just a few basic daemons like Syncthing used in mint cinnamaon. 4GB is pretty tight unless you are willing to make some behavioural changes or use a less friendly distro. But 8GB is more than enough.
Different story trying to run VMs on my server, though.