The “all” stream would be all of the posts from the combined subs of the users on the instance. So if there’s a community nobody is subscribed to, it won’t appear on all. This is true of all instances. Many smaller ones will employ bots to crawl Lemmy and sub to communities to give the large instance “all” feeling.
That being said, yeah it’s all preloaded onto your local server. There is no difference in speed. Doesn’t matter if it’s active/subed or new/all they all load the same
I’d highly encourage everyone to find smaller instances and leave lemmy.world for the immediate expats. Find something that aligns with your values. Or if you are technically literate enough host your own instance. If you have an old desktop computer you’ve already got everything you need.