I don’t think there is any particular problem with people having fewer children. It creates a short-term labour burden in terms of having to divert more resources to elderly care but the rate of birth change isn’t likely to change too dramatically to affect much. You just reduce the population over time and that’s fine. Fewer people isn’t a problem as long as the labour exists to support the food, clothing and shelter output you require for the combination of capable + incapable that your community has. Starvation in the short term is the larger concern.