I understand your math but I think the better view would be “what are the monthly payments at each point in time (no inflation) and how many of hours of work per month based on the median income of the time?”.
Based on that the 1980 house would have a payment of $731/month. Median income was $21,020, which is about $10.11/hour. So it would take about 72 work hours a month to pay for a home.
The current rates yield a monthly payment of $3,107. Median income is $56,940, which is about $27.37/hour. So now it would take 113.5 hours a month to pay for a home, an increase of almost 57%.
To make it even worse, there are usually only 160 work hours a month, so that means you have to work 71% of a month just to pay for an “average” house and mortgage. I’m not even factoring in taxes or any other expenses.