Alternatively, buy a plot of land from an area less desired by the general public. Have the house built yourself, and move in. House buying is not smart unless you are pressed for time, which is not a smart way of handling it as you should take your time with houses.
Buying a house or land in an area that is saturated with demand is also unwise. Live within your means. Most who are able to afford land and house are more than capable of affording more time away from the job or remotely work. If your job requires a lot of your time, then it is more wise to just rent a small room and take vacations outside the area. There are a lot of factors in place when you consider a house, and you made great examples.
I am sorry about the interest rate. A 6.5% rate is not the worst but disadvantageous because, historically, most ROI for safe investments are 5%. This means it is more beneficial to pay off the house than it is to diversify while you are paying the house. Which is bad because you should be diversifying your portfolio. My sister has a 2% interest rate, she is so lucky and I tell her to diversify instead of putting all her money towards paying off the house.