As someone with diagnosed mental health issues, I can tell you that I mean no harm. But sadly, the pure difference in perception paired with unfamiliarity of the two parties makes the situation insanely dangerous, not the person.
You never know why the person is in this predicament and if they have a tendency towards violence, robbery or other things. I‘m not saying they are. I‘m saying you have to assume they are before making that decision. Can you defend yourself against a pulled knife or even gun, do you have enough mental capacity to observe them at all times. Those odds do not look good.
So, although I would never willingly look down on folks less fortunate than myself. I too will never ever let a stranger into my house if they raise any concerns.