Generally, I agree with this. But not with opioids. They are far more dangerous than most other types of drugs, for a litany of reasons. Even under a doctor's supervision, there is basically no "responsible" use of these drugs. Most people who become addicted to opiates do so after receiving them from a doctor in the first place.
If even doctors can't administer these drugs safely, then neither can Joe Schmoe trying to score a hit of fent.
I'm a recreational drug user. Admittedly an addict to some of them. So are many of my friends and family. I'll dabble in some coke or molly or LSD, but opiates are far, far past the line for me, and should be for any sane user.