If immigration leads to more unemployment, then that is an economic problem, especially in the hypothetical case where the social benefits system is getting more and more strained by an influx of unemployed people. But generally, I think that you can expect that the immigrants will soon find employment. Besides that, there’s the cultural aspect that @jet mentioned. You could also make the point that the country’s infrastructure is more and more stressed as the population grows, but that is fixable and potentially counteracted by the labour potential of the immigrants themselves (i.e., qualified immigrant work forces can make a large-scale infrastructure overhaul possible that will lead to greater national capacities and a net benefit for the entire population).
Aside from these things, I would argue that most of the other reasons boil down to xenophobia or racism.