Yeah you nailed it! It’s generally not a problem at the primary colour level. For example, I have no trouble distinguishing the red and green of a traffic light. But there are, like you say, a lot of ways to mix colours and that’s where we get into trouble.
There is a type of colourblindness that affects your perception of yellow, but the red-green types are far more common in the population. It is also much more common in men than women. This is because the trait is carried by X chromosomes, and for a woman to exhibit it, she must have the gene on both X’s. Men only have one X and one Y, so if you have it on your X, you’re doomed! lol
One thing you may or may not have noticed is colourblind people tend to be more sensitive to subtle differences in shade. This is a compensation the brain works out, similar to how fully blind people develop a sharper sense of hearing. The military even noted that this can help in spotting camouflage on an air photo, and it’s one of these rare cases where colourblind individuals were specifically sought out. It usually goes the other way, though. Like we are steered away from careers as electricians, pharmacists, etc. and I totally get that. It’s not discriminatory. It’s just common sense! :)