An issue is that lower income areas often have less focus on things like test taking skills, so genuine ability is really hard to distinguish from test taking practice.
Also, schools in lower income areas often aren’t nearly as good, forcing a cycle of poverty since they can’t get into college very easily at all.