And it’s not like the output is saved for the next time; you need to do it every time.
You can cache transcoded content in Jellyfin. So use a large enough cache and you basically only have to transcode once for every resolution. It's easier for me to set up transcoding than it would be to manually figure out which resolutions I'll prefer having around and transcoding them. Most of my stuff exists in 1080p, with 4k files for stuff I REALLY like, but I sometimes find myself watching on very low resolutions on my phone when away because I have pretty limited data.
I find that in a few movies the 4K versions have a generally better image quality and are worth it even if you are sitting far away or not watching the content in 4K resolution at all. But like you, I only keep around 4k files for stuff I really like.
EDIT: I've also run into problems with codecs on other people's devices when not transcoding. I could keep my files in whatever the most compatible codec is nowadays but having the ability to transcode on the spot is easier.