I find the OP’s question very intriguing and have kind of arrived at this same conclusion. My only tweak would be that they may, in fact, have more effective immune systems purely due to the fact that access to medicine or areas free of pathogens aren’t as common. Obviously, though, that would be compared to a person who exists in those same conditions but with access to good medical care which is a bit paradoxical.