You have to feed and house the people. The people currently living in those countries may have a shortage of housing already.
I think the “baby boom” from years past has shown that there are too many people around. It’s too costly to raise their own kids. People are xenophobic and don’t want many immigrants changing their cultures.