Yes, lack of housing is the bigger issue. Here in the Boston area, it’s pure supply and demand. Neighborhoods are full of triple-deckers just off the city center that could be denser apartment buildings. Landlords can charge whatever they want, because they know that anyone who wants to live in the area will have a hard time finding another place.
There’s also the question of transit infrastructure. Even with less dense housing, if there were easy ways to get around other than cars, proximity would be less of an issue.