Highly customized/optimized Linux images certainly are one use case of gentoo.
if it’s cool you might be willing to put up with the drawbacks
The “cool factor” is a significant point. My gentoo laptop (which I update rarely besides browser/security updates) boots in under 3 seconds to graphical login :-)
Compiling can be done by a the cluster
Actually most compiling is pretty quick on modern systems (compile in DDR4 ramdisk, nvme, fast CPU etc.) I’d say, most stuff compiles as quickly as installing a binary nowadays.
It’s the huge stuff that’s annoying: webkit, rust, Qt, boost, firefox/chromium etc. But one can skip updates easily or use precompiled binary packages that are provided for big stuff.
Pi4 is perfectly doable. But Pi Zero won’t be a lot of fun.