In the United States at least, your local government’s public hearings for new housing developments kinda begs to differ.
People will demand the homeless be eliminated from their area while simultaneously opposing development of housing or shelters for the homeless in their area.
So maybe you’re right though: they don’t hate the apartments more, they simply can’t make up their mind on which they hate more.