Not collecting protected class data does not prevent algorithmic bias
Adam Wenchel · Production AI: Lessons Learned the Hard Way // Adam Wenchel, Arthur.ai (FirstMark's Data Driven NYC) · Jan 22, 2020 · at 6:56
Adam Wenchel, CEO of AI monitoring platform Arthur.ai, discusses modern machine learning compliance and governance requirements at Data Driven NYC in January 2020.
0:00 / 0:19exact quote · 19.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“What's happened a lot in the past is people have kind of like taken the head in the sand approach where they've sort of said like, oh you know, we're not even collecting protected classes, so we can't possibly be biased as far as we know, and that's no longer good enough. You now actually need to actively kind of monitor for bias and make sure that your data scientists are incorporating notions of fairness in the algorithms.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Adam Wenchel
Most companies using AI suffer from unpublicized model failures
“Every company that's doing anything substantive with AI probably suffers from any of these problems, it's just a few of them have actually, ah, made the headlines for it”
Open-source explainable AI tools like LIME and SHAP fail enterprise scale
“If you look at the open source components they're not very scalable. They're not easy to deploy at scale. They're really, they're useful, like, if you're a data scientist, and you have your Jupyter notebook, and you're, you know, running an experiment locally,…”
Real-world AI deployments introduce distinct failure modes beyond lab environments
“And it's not only hard to develop in the lab, but once you develop it and put it in the real world, there's a whole new set of categories of ways it can go wrong.”
Historical anti-bias regulations apply directly to AI models
“There's a lot of historical regulation around anti-discrimination and bias and things like that that, ah, certainly applies just as much to AI models as it does to humans and more simple analytical models.”
Lack of trust delays enterprise AI deployments for months
“We, you know, encounter this all the time, where in organizations, they have these big plans for AI but they're just, They're unsure about actually deploying them and turning them on, and things get held up for months and months and months because of that.”