At the risk of sounding crazy… You might enjoy Linux From Scratch (LFS) and Beyond Linux From Scratch (BLFS). Maybe not as a daily driver, but it’s a great way to learn how everything works, since (as the name suggests) you build everything from scratch. No package manager, just tarballs of source code. It really helps with getting an understanding of how everything works.
BLFS even includes instructions for building Xorg and all the major desktop environments.