The problem is just over 50% of our population seemed to be wearing rose-tinted goggles of back when we controlled over half the world's trade...
They failed to realise the reality that we relied on the EU as much as it relied on us, and cutting ourselves off from our primary trade partners was a plainly stupid thing to do.
There was also the racists who saw it as a way to get rid of the "foreigners", not realising many of these immigrants weren't from the schengen zone, therefore wouldn't be "sent back", but that's a different story.