Well, you are not alone. While I too would prefer not to use proprietary drivers, I have had no problems on any of my Nvidia machines as well. Ironically, despite the open source drivers, getting a 7900XTX card up and running was an issue for me for months till distros caught up (with newer kernels and mesa libs), while my 4090 installation was a breeze even on the day it was released.
A lot of problems people have with Nvidia GPUs seem to be installation related. I think that is because the installation tends to be distro-specific and people do not necessarily follow the correct procedure for their distro or try installing the drivers directly from the Nvidia site as they would on Windows. For example, Fedora requires you to add RPMFusion, Debian needs non-free to be added to sources, Linux Mint lets you install the proprietary drivers but only after the first boot, and so on. Pop OS! probably makes the process the easiest with their Nvidia-specific ISO.