Except that somewhere down that chain someone is almost certainly going to choose to kill people, so by passing the trolley on down to them you're responsible for killing a lot more than if you ended it right now.
And since every rational person down the line is going to think that, they'll all be itching to pull the "kill" lever first chance they get. So you know that you need to pull the kill lever immediately to minimize the number of deaths.