If “no, thanks” is not treated as a complete sentence, you’re in a bad crowd. Doesn’t matter if it’s beer with the coworkers or MDMA at a trippy cuddle party. “No” requires no further elaboration.
If you feel like discussing your reasons, feel free to bring them up. But you should not have to. If your “no” is not accepted about drugs, get out – because those people are in the habit of rejecting people’s “no” on other things, too.
And by the way, this rule is just as important, maybe more so, for people who do choose to take recreational drugs sometimes. Just because I’m okay having a beer in one context does not oblige me to do shots with your buddies.