There are easy ways to have a bad time using psychedelics (like not picking a controlled environment, not being prepared for that your trip will take some time). Knowing these things ahead of time/being prepared matters quite a bit in terms of your ability to have a safe, pleasant trip.
This sort of knowledge, sort of like “you shouldn’t plan to operate heavy machinery after drinking that cough syrup or those 7 beers” is key to responsible use- and it’s the irresponsible users that become the poster children for the ‘ban everything’ crowd.