Let me spoil part of the Foundation series for you. In one book, the cast visits a planet where they encounter one person with psychic powers surrounded by robot servants. He reveals that the planet is evenly divided by I think 128 people like himself who want for nothing and live comfortably. They only reproduce asexually, and only in preparation for their own death or when another dies.
What this illustrates that’s relevant for you is that yes, not hitting the replacement rate could lead to significant population decline, but only until people are comfortable enough and want to have kids or feel it is the best way to maintain their way of life (think farmers having kids to help on the farm).