Gravity bends light. You can literally detect large masses by how they distort the space around them. From what we can observe, space overall is pretty flat. AND we can measure where stuff is going using various proxies i.e the doppler shift, spectral lines, standard candles etc. If stuff was being pulled toward some mass, its going to cause deviations in how fast things are moving relative to each other. Which in the case of the great attractor, is what is happening. But if you are asking whether everything is falling toward something in every direction equally, the answer is no due to the shell theorm.