In the general public I don’t think it’s pure xenophobia, but in large parts a fear of not being able to communicate. The behavior they expose then is almost indistinguishable from xenophobia and for the receiving person in no way better, though when a foreign looking person is able to communicate, most hesitations will fall away.
I know that this is obviously not applicable to 100% of the population, but so is the other belief. There are obviously also die-hard foreigners hating nationalists, as in every country.