A mortgage is an investment, let it roll for as long as you can.
The fact that you're not wrong about this is part of the reason housing keeps getting more expensive. So many individuals have their nest eggs tied up in their houses that building more would hurt a lot of ordinary folks.