Why should someone who is already breaking the law also enjoy the power of legal coercion to force you from a place you had the legal right to be, though?
“Well, we don’t want the situation to escalate. Someone could get hurt.” Why should the law protect only the welfare of criminals? Of the person actively breaking the law?
The issue with “Stand your Ground” laws is that the alternative is nonsensical if your view expands to include the rights and welfare of people who act consistent with the law.