These drugs are typically used for individual and group therapy sessions and not given as prescriptions to take at home. They’re basically just there to augment therapy sessions.
I do think they should be legalized and allowed for recreational use as well but your concerns aren’t really justified if you look at what existing therapeutic applications are like and what companies in the psychedelic space are trying to do right now.