In order for off-instance content to appear in search (or at all) it has to have been federated, and only communities that have at least one subscriber, are federated to the local instance. If some other community, on another instance has no subscribers on your instance, it doesn’t even appear in “All”. To get that first sub, someone just has to “know” and write it’s exact name in search, for your instance to reach out and establish federation to the off-instance community. Most apps can’t even do that step, you have to do it via the webUI, though I know Thunder can do it since a recent update.
The reason an instance can’t search the entire fediverse, is that it doesn’t know about the entire fediverse.
External web pages like lemmyverse.net don’t either, but unlike instances they go out looking for what’s out there, rather than only connecting to communities at least one user has told it about like the instances do.
The idea is to only cause federation traffic between instances where its actually wanted. For a truly fediverse-wide search, every time you search, your instance would also have to ask every other instance for search results, then return and compile those thousands of results from thousands of instances. It’s not a sane way to do things.
In the future, it’s more likely that instances or clients will connect to “search providers” like lemmyverse that maintain indexed databases of the fediverse, and be able to get results without hitting the actual network with a search query at all. It’s possible some form of more advanced indexing will become a part of the instance server software itself.
But vry little work around this has been done, but you can see it in places. Thunder, the client I’ve worked on, only has basic search right now, but there has been talk about solving this problem within the app, eventually, by providing results from lemmyverse.