When you’ve only seen one in your life, you can differentiate them from everyone else using some broad attributes. You don’t have to pay very close attention to the nuances of their facial structure.
It’s only when you know a lot of people who all have those broad attributes that you have to start noticing other things.
This isn’t racist, it’s just the brain doing what it does best: not work harder than it has to.
It sounds like you’re already doing the work of looking twice, looking closely, paying more attention. Just keep doing that. And remember you don’t have to feel bad about this.