I once swapped a Debian install with XFCE to just running Openbox instead of a full DE and got down to 300Mb or so of memory usage. This was about a decade ago so obviously YMMV but given literally all I did was run Debian with just openbox and no DE, there’s probably additional tuning to be done that can get them to a more usable state