Most of the wealth was from trade. That’s how the empire started. Money was made from trade and the industrial revolution then came empire. I guess you can say the empire started from stealing money off the Spanish but they didn’t exactly follow us home.
Empire and mass imigration aren’t related, unless you are talking about how brits spread to North America, Australia and New Zealand. But you aren’t. Lot of men died in WW2 and labour was needed temporarily to rebuild and man the factories, there was a lot of damage in ww2. The government realised it is good for business to have cheap labour so kept the workers. Then they realised wages can be suppressed, houses prices and demand for other things can be increased, as well as gdp by importing loads of people from the third world.
By an huge amount most of the immigration has come post empire when the countries are independent.
Also the empire was expansionary in nature. It wanted to build Britain abroad, infrastructure, law, order, “civilisation”. Many countries got it’s independence through a vote, I think only two ever declared independence and one is very friendly with UK wouldn’t be more surprised if people leave the UK more than they get back, and the other country doesn’t exist anymore. Britain was never about importing people in, it was about exporting people and civilisation out.