It's kind of like in the 1800s they believed that human beings could never generate enough pollutants to actually affect the Earth. There are intelligent opinions statements saying that the Earth was simply too large for anything a mere handful of 100 million humans could do to leave any lasting impact on.
Of course, they had no idea that we would swell to 8 billion humans or that the industrial revolution would take off quite so well as it did, but even today there are many people who believe that nothing that they individually do can leave any type of lasting ecological impact, positive or negative.
And because of that you have bum fuck HVAC technicians venting refrigerants into the atmosphere willy-nilly and assholes driving down the street throwing lit cigarette butts out in the middle of a drought and people just dumping their trash wherever they find an opportunity to dump it.
I said all of that to say that it's probably likely that even minor usage of drugs cause effects that are at best difficult to quantify. I don't think getting high one time is going to be the differentiation between a homeless bum and a Nobel Peace prize winner, but it might be the difference between someone who works a career and earns at their best $250,000 a year and someone who works a career and earns at their best $80,000 a year.