They increase the overall cost of both buying and renting a property within that market, and are a nuisance for existing residents.
Historically – in the UK, at least – the market equilibrium has been that the rich own all the property and the poor pay rent until they die, aware that they can be served an eviction notice at any time.
This has not proven to be a popular policy. In 1918 all British men, regardless of whether they owned property or not, got the vote, and since then politicians have found it useful to not have the majority of voters perpetually furious about it.