There is definitely some substance behind the complaints, but I think they are overblown or just the typical linux-user-parroting something they heard other people say.
On PopOS my 3070ti was always stable. I ran into occasional stuttering in the DE, but the biggest thing was I had to manual compile shaders using some guys github repo to play Apex Legends without it being a stuttery mess. But like you said, Pop is on X11 so maybe that makes a difference?
I bought into the “if you are going to use linux, especially for gaming, you need an amd gpu.” So I bought a 6900xt. I’ve had as many issues with my 6900xt; they are just different types of issues. Nothing insurmountable but its not like its some panacea.