In NYC, companies will have to prove their AI hiring software isn't sexist or racist

nodsocket , 11 months ago

One easy way to prove that the AI is racist is whether race is one of its parameters. However, it is possible for an AI to be racist even without an explicit parameter.

reilwin , 11 months ago

Not necessarily, something that overt would be obvious and avoided. I’m pretty sure that what they’re looking for is more for subtle biases caused by bad datasets to train the AI.

For instance, you train the AI and tell it which candidates are good or bad. But maybe, by pure happenstance, the best candidates in your dataset are all male. If so, the AI might be accidentally trained to believe that all good candidates are male.

HobbitFoot , 11 months ago

Or you train the AI based on data from humans who are racist. If an AI learns how to hire from a racist set of human managers, it may take those biases into account when it is choosing who to hire.

DreamerOfImprobableDreams , 11 months ago

I remember hearing about a high-profile case where the AI would dock points if someone's resume listed them as participating in women's sports as an extracurricular, while giving extra points if it listed them as participating in men's sports.

Also, bias doesn't necessarily have to come from happenstance. Unfortunately, humans tend to have unconcious (or, sometimes, not-so-unconcious) biases against women and people of color. There was a study where researchers sent identical resumes to a random group of recruiters-- but half of the resumes had a male name and half had a female name.

They found that both male and female recruiters were more likely to rate the resumes with the male name higher and be more likely to recommend they be advanced to the next round of interviews. IIRC, similar studies have found similar results if you give the resumes a "Black sounding" name versus a "white sounding" name.

So if you train an AI on your own company's hiring data-- which is likely to be tainted by the unconcious bias of your own recruiters and hiring managers-- then the AI might pick up on that and replicate it in its results.