The tribe was certainly larger than 8 people, but the tribe would also require regular births to growing. And in prehistoric times there was a very high mortality rate for children. And the only two ways to combat that is a)provide safer environments for the children or b)have more kids.
A) wasn’t an option since they didnt have the means to, but b) was so long as you had enough fertile women. So losing 4 men is a serious blow into the productivity of the tribe, losing 4 women to a tribe struggling already means 4 less potential births next year. You have 20 men and 1 woman, you only have 1 potential birth in the next 9 months. 20 women and 1 man, you have 20 potential births over.
Child rearing was the only thing women could do, but it was easily the most important thing to the future of the tribe. All other things being equal, the men were more expendable than the women.