I think the biggest issue is religious people that can only view things through the lense of their own perspective equate any belief system with religion. And since we, as humans, categorize everything, everyone has a “belief system,” even if you believe in absolutely nothing (nihilism).
So “atheism” is considered a religion, believing in the scientific method is a religion, and believing in the inherent and equal rights of people is a religion.