NYC Wants to Ban Employers From Using Automated Hiring Tools to Curb Discrimination

A bill passed by the New York City Council this month would bar employers from using automated hiring tools unless the tools can pass a yearly audit to prove they don't discriminate based on a potential employee's race or gender, the Associated Press reported.

While some companies use artificial intelligence-based tools to review resumes or analyze video of job interviews, applicants usually remain unaware their information is being put through the process. The bill would also force the companies who create the AI tools to provide more information about how their products work, and give candidates the chance to opt out of having their information reviewed by a program, potentially requiring a real person to go over their application.

"I believe this technology is incredibly positive, but it can produce a lot of harms if there isn't more transparency," Frida Polli, co-founder and CEO of New York startup Pymetrics, which uses AI to assess job skills through game-like online assessments, told AP. Her company lobbied for the legislation, which favors firms like Pymetrics that already publish fairness audits.

However, some AI experts and digital rights advocates said the bill won't go far enough to prevent bias and sets a relatively weak standard for a bill that has the chance to be used as a template in other states, or federally.

The bill comes as more companies like fast-food chains and Wall Street banks have begun to use AI tools to speed up their application and hiring process.

The council passed the bill 38-4 on November 10, and while Mayor Bill de Blasio has said he supports the bill, he is still unsure if he will sign it. The bill has enough of a majority to go into law without the mayor's signature, and once enacted it would go into effect in 2023 under Mayor-elect Eric Adams.

For more reporting from The Associated Press, see below.

Proponents liken it to another pioneering New York City rule that became a national standard-bearer earlier this century—one that required chain restaurants to slap a calorie count on their menu items.

Instead of measuring hamburger health, though, this measure aims to open a window into the complex algorithms that rank the skills and personalities of job applicants based on how they speak or what they write.

"The approach of auditing for bias is a good one. The problem is New York City took a very weak and vague standard for what that looks like," said Alexandra Givens, president of the Center for Democracy & Technology. She said the audits could end up giving AI vendors a "fig leaf" for building risky products with the city's imprimatur.

Givens said it's also a problem that the proposal only aims to protect against racial or gender bias, leaving out the trickier-to-detect bias against disabilities or age. She said the bill was recently watered down so that it effectively just asks employers to meet existing requirements under U.S. civil rights laws prohibiting hiring practices that have a disparate impact based on race, ethnicity or gender. The legislation would impose fines on employers or employment agencies of up to $1,500 per violation—though it will be left up to the vendors to conduct the audits and show employers that their tools meet the city's requirements.

Julia Stoyanovich, an associate professor of computer science who directs New York University's Center for Responsible AI, said the best parts of the proposal are its disclosure requirements to let people know they're being evaluated by a computer and where their data is going.

"This will shine a light on the features that these tools are using," she said.

But Stoyanovich said she was also concerned about the effectiveness of bias audits of high-risk AI tools—a concept that's also being examined by the White House, federal agencies such as the Equal Employment Opportunity Commission and lawmakers in Congress and the European Parliament.

"The burden of these audits falls on the vendors of the tools to show that they comply with some rudimentary set of requirements that are very easy to meet," she said.

The audits won't likely affect in-house hiring tools used by tech giants like Amazon. The company several years ago abandoned its use of a resume-scanning tool after finding it favored men for technical roles—in part because it was comparing job candidates against the company's own male-dominated tech workforce.

There's been little vocal opposition to the bill from the AI hiring vendors most commonly used by employers. One of those, HireVue, a platform for video-based job interviews, said in a statement this week that it welcomed legislation that "demands that all vendors meet the high standards that HireVue has supported since the beginning."

The Greater New York Chamber of Commerce said the city's employers are also unlikely to see the new rules as a burden.

"It's all about transparency and employers should know that hiring firms are using these algorithms and software, and employees should also be aware of it," said Helana Natt, the chamber's executive director.