And I think it’s a result of our evolutionary trait of being very sensitive to people looking at us. If we see someone out of the corner of our eye looking at us it alerts us. And if we truly can’t see them, we can’t tell at all, but checking to see if they’re looking at you makes them feel stared at, so they’ll look at you back. Is that confusing?
Wikipedia sums it up well. In proper scientific tests, it’s fruitless.