In that case I’d go for something very barebone. Get a minimalistic debian up and running, and see how that works. There are plenty of lightweight desktop environments (to the extent some of them count as desktop environments), where TWM is an extreme example.