It’s not fewer people that’s the problem, but fewer people too fast. A society needs labor to provide the goods and services people need. If the share of people who do labor (working age) to people who don’t (children and the elderly) becomes too lopsided, the burden on those who work becomes unsustainable. (The Boomers had the opposite: they had a smaller older generation and didn’t have many children, so during their prime years the working age population was much larger than dependants on both ends of the age pyramid. That’s part of the reason why they were so prosperous.)
Going by total fertility rate (children per woman):
2.1 is enough for replacement. No problems.
1.8 means every generation is 10 % smaller than the previous. We can deal with that.
1.5 means every generation is 25 % smaller than the previous. This starts to cause problems.
1.0 means generation size halves every generation. This is not sustainable.