Honestly, the ignorance goes both ways. I used to think very lowly of religious people in general and maybe especially Catholics back when I was a teenager who had just realized the profound truths of atheism. Having travelled a bit more and made friends from various religions, I've realized that while my diagnosis of the Catholic church was somewhat accurate, judging the people was a product of my own ignorance rather than theirs.