From an outsider perspective (I haven’t used Nix at all), the downsides I see are that it’s extra software on top of the defaults for any given distro, it’s not optimized for the distro (meaning it might pull in dependencies that already exist or not use distro specific APIs/libs), and it doesn’t adhere to the motivations of the distro (e.g. not adhering to the DFSGs for Debian).
And of course, most of the packages are community maintained and there’s the immutability, which might be a hinderance to some use cases, but not for me.
All in all, not really the worst if you’re not worried about space or getting the absolute most in performance and not an ideologue, but it’s enough to make me stick with APT. I chose Debian because of its commitment to FOSS, not the stability nor performance.