Freedom of movement never was and never will be a thing outside of countries with similar standings in economy and policy.
There’s the obvious problem #1) People rushing to whoever maximizes their welfare. There’s this fine reason why plenty of illegal economic migrants do not settle for some first-world country that accepts them and keep going until they hit something like Germany.
Then you have #2) Societies do not exist without a place and no society should be forced to accept people that undermines it. France is secular and yet it allowed in plenty of people that are not. I’m not saying you must be secular to exist; I’m saying that you should not be going to a society you fundamentally disagree with and much less start imposing. And yet we both know what would happen if borders were open.
You also have #3) rich people can just buy out the nicest places and chop chop people the fuck out. A state putting up some barriers severely slows this process (which is happening anyway)
A bunch more reasons like paperwork, criminal record, ecology, yadda yadda.
With this said, if you fulfil stuff, you should definitely be able to get wherever you want. Ethnicity, social status ou whatever made up stuff should not be roadblocks. Even if it takes a year or two of screening and some sort of integration procedure.