Despite space being “empty” there’s still a surprising amount of stuff streaming through it. There are protons, electrons, carbon nuclei, etc constantly slamming into the Earth’s atmosphere, producing showers of radiation. These cosmic rays are the reason so many sensitive physics experiments ( like dark matter and neutrinoless double beta decay searches) are located deep underground. The earth is a good shield against these cosmic backgrounds.
Even if there was an “isolated” antimatter galaxy, it would get bombarded with matter in the form of cosmic rays. The annihilation photons are a really distinct signal that would be hard to miss. There are a number of gamma ray telescopes in space that map out sources of gammas, and they would have detected an antimatter galaxy if it existed.
If the antimatter galaxies are so far away that they’re beyond the visible universe, then there’s still the big question of why there was a segregation of matter and antimatter early on.