The concept of whiteness as being a distinct and unifying factor among people from a variety of countries led to the field of scientific racism, which went on for around 2 centuries. This is where terms like "Caucasian" came from. Scientific racism was used as a reason by the Nazis to murder millions of Jews. Now in the 21st century, we know it is pseudoscience even though its terminology lingers.
This doesn't mean that people were blind to skin color before 17th century colonialism, as you pointed out with the Indian caste system that dates back as far as 3300BC. You can see in ancient Egyptian art a wide array of colored skin as well. From Egyptian literature, there aren't signs they really obsessed over it the way society does now.
I think we can all agree that things didn't go too well for non-white people after the invention of whiteness. The concept was founded in oppression and helps perpetuate oppression.