Yeah, I think you're right, and I think that's exactly why it's a blind spot for me.
On several occasions I've also lent an old laptop to friends when theirs broke, and all of them ended up using Linux for months no questions asked. They later went back to Windows because of the Word grammar check, but other than that it just worked for them.
But of course, if you can't get your drivers to work it'll be a completely different experience.