Africa’s climate doesn’t offer that much deadly cold.
And where caves were involved, they may not have been a constant dwelling but a place to retreat to in times of need.
Your spelunking friends don’t consider rain a deadly threat because they have REI jackets and access to antibiotics, so their attention is entirely on the issues of rain destabilizing the cave. But disease and exposure were immediate deadly threats to more primitive people.
So hiding in a cave during the rain, despite your spelunker friends’ modern safety standards, was probably a survival tactic.