I'm not personally colorblind, but lots of men on my wife's side are, including my son. My understanding is that it doesn't affect brightness, it affects being able to differentiate colors.
It's important to note at this point that we're talking about mixing dye or paint colors, which behaves differently than mixing colors of light. When you mix red and green light, you get yellow. When you mix red and green paint, you get brown.
So to my son, for example, when you have an object with a mixture of mostly red and a little green - I would see that as "mostly red with a little green," while he would see it more like "brown." My expectation would be that if he was in an otherwise dark room illuminated with only red light, that he would see objects with a similar clarity as I would, but that his experience of the color would be different from mine in a way that I could never really understand (and vice versa).
Since I have access to a relatively large number of colorblind people, this makes me want to do an experiment.