It should be said that packages usually don’t work like this however (they usually depend on one central thing which then depends on some proprietary thing) and if they do, there’s usually an override setting for the package in question with the dep already wired up.
What you’re actually looking for here is a Gentoo USE-flags like system. We have a precursor of that in i.e. config.cudaSupport but it’s not widely used.
how would you add JPEG-XL to ffmpeg
You’d do that upstream because that’s something that every instance of ffmpeg should be able to do. And also ping me because I maintain ffmpeg ;)
and other packages that make use of it independently (e.g. so that Gwenview supports JPEG-XL)?
I’d expect most packages to make use of such a feature dependently. I don’t know how gwenview or QT stuff works but from a quick Google I gather that KDE stuff has the kimageformats library which then in turn depends on jxl. Gwenview should depend on that but only depends on qtimageformats currently? I’d give adding kimageformats to its buildInputs a spin if you’re curious. If it works, make sure to submit a patch against Nixpkgs.
This isn’t the sort of thing you should solve in an overlay (better just do it upstream) but you certainly could.