Because they don’t like other people being around, it’s a simple choice if you want to live somewhere quiet then you move somewhere quiet you don’t have the option of telling everyone else that they can’t enjoy a beauty spot or city.
Especially when plenty of other people there choose to be there because it’s popular.