AI hiring tools may be more biased than we thought, in ways we didn’t think

New AI-powered screening tools used by companies in their recruitment process could pose a massive problem for fair and equitable decision-making across the entire hiring landscape, according to findings from Stanford University.

A research paper published last month by the Stanford Institute for Human-Centered AI warns that widespread use of AI-powered hiring tools may exacerbate biased outcomes and discrimination across entire sectors as companies flock to the same tools and vendors to aid in the hiring process. An algorithmic rejection by one company will likely be duplicated at another and another and another if the companies are using the same hiring software to screen candidates.

“AI screening tools bring together three properties that should not co-exist in high-stakes decision-making: they are pervasively adopted, highly consequential, and opaque to the public,” the study’s authors said.

At issue were two findings: First, researchers found that 26% of Black applicants and 15% of Asian applicants applied to positions where they were discriminated against on the basis of race by the AI system of pre-employment screening company Pymetrics, which gamifies the screening process using neuroscience-based mini games to evaluate candidates, eschewing the traditional résumé and cover letter. The study analyzed Pymetrics data from nearly 4.2 million job applications submitted by 3.4 million applicants to nearly 1,800 positions across 150 employers in 11 sectors.

The researchers investigated bias on a per-job basis rather than a single-employer study, which lacks specificity at the job-level. If an AI tool over-indexes for Black applicants applying for warehouse roles but infrequently recommends Black candidates for corporate roles, like in finance, those two patterns could cancel one another out and the discrimination might be missed by an employer’s bias audit. Aggregated bias audits published by the vendor report no measurable bias, according to the authors, demonstrating how unmeasured the implications might be.

The findings were specific to the Pymetrics dataset, an obvious limit of the study’s findings (it’s also worth noting that the vendor leverages game play in the process, which is a popular but not dominant screening mechanism), but more concerning to the researchers is what they’ve termed “algorhythmic monoculture.”

The World Economic Forum estimates that 90% of employers deploy automated screening systems in the hiring process, and when a small handful of vendors help a majority of employers make decisions about who to interview or ultimately hire, bias findings apparent in the algorithms will be repeated across the world of work. The scale of the problem could be massive.

“This suggests market concentration matters: As a single hiring vendor comes to dominate screening for an industry, it may be more likely that candidates are shut out,” the study’s authors concluded.

The findings come as a number of vendors face legal scrutiny over what’s inside that “black box” screening algorithms use to deliver recommendations to recruiters. Workday is facing a massive class-action lawsuit claiming its AI-powered applicant screening and recommendation tools discriminate against certain types of job seekers. Eightfold AI is facing a different lawsuit over allegations its AI-powered services to employers illegally mirror those of credit agencies, except without any of the protections afforded to job seekers under current consumer protection laws.

HR leaders and recruiters are monitoring those cases as they work through the court system. The outcomes could indicate to employers how the legal system is approaching AI litigation.

“I don’t think we want to discourage the application of AI in this domain, but recognize the stakes are high and be judicious in the approach,” Rishi Bommasani, one of the study’s authors, said in an interview about the report.