If the share of people who do labor (working age) to people who don’t (children and the elderly) becomes too lopsided, the burden on those who work becomes unsustainable.
Except that raising children requires more time and resources than caring for elderly. So having less children frees up more resources to care for the elderly. Into the next generation there are now less people which require even less resources which means you need fewer workers to produce those resources.
History provides evidence for this. After every major war there were economic booms. This is despite wars killing off the able bodied workers leaving only the sick and elderly.
The only people who suffer from a lower population are the ownership class. They live by skimming a little of the productivity off of every worker.