It’s because the question is weaponized. It makes the assumption that just because you don’t look like me that you can’t possibly be a “real” American. And asking the same in reverse doesn’t work, because white people in the US love saying where their ancestors are from.