Because different castes are generally different ethnic groups. Most people can tell Chinese, and Japanese people apart just by looking at them. You can do the same.
I’m not Indian, I’m Pakistani and while there isn’t a caste system in Pakistan, there is definitely discrimination based on ethnicity in Pakistan.
And while most people can’t tell Pakistani people from each other in the west, Pakistani people themselves can look at each other and tell if one is Muhajir, or Siraiki or Pathaan or Sindhi.