Not having something in the first place and losing something you have are two different things. It’s like saying to someone who just lost their partner “don’t feel bad, for the first n years of your life you didn’t have a partner and you were fine”
Additionally, it’s not billions of years of nothing. It’s an eternity of nothing. Billions of years may as well be the blink of an eye relative to eternity.