I think you need to let go of this idea that the big bang happened at a specific location or that we are moving away from or towards a point in space. There is no specific point from which all points in space are departing. Instead, every point in space is moving away from every other.
Consider the following!
Point at any object. That object is at some distance away from you in space which is also a distance away from you in time. For example, an object one light year away is one year into the past, as perceived by you.
Now, point in any direction. No matter where you point, if you extend the line forever, it intersects with the big bang. There is no place to which you can point, no direction, no infinite line you can draw that doesn’t include the big bang.