The portal is a hole. The hole is moving. The conservation of momentum is the hole moving as it continues to move along the track. If the people start moving, where does that momentum come from?
Imagine a tennis racket with no strings. Two portals are stretched across the space the strings would normally be, back to back, one orange one blue. If you threw a ball in the air as if you were going to serve and swung the racket, the ball would pass straight through the portals as if they weren’t there and would fall straight down due to gravity. The ball maintains its conservation of momentum, and the tennis racket holding the portals also maintains its conservation of momentum as it swings through the air. There is no force applied by a hole.