A fractional drop in AI performance can cost hundreds of millions
Adam Wenchel · Production AI: Lessons Learned the Hard Way // Adam Wenchel, Arthur.ai (FirstMark's Data Driven NYC) · Jan 22, 2020 · at 8:46
Adam Wenchel, CEO of Arthur.ai, speaks at Data Driven NYC about why enterprise machine learning models require continuous monitoring in production.
0:00 / 0:12exact quote · 12.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Even if your model encounters some sort of issue that drops at a couple 10th of a percent in performance, that can literally be hundreds of millions of dollars over time, and so the ROI on having that kind of monitoring in place is, is huge.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Adam Wenchel
Most companies using AI suffer from unpublicized model failures
“Every company that's doing anything substantive with AI probably suffers from any of these problems, it's just a few of them have actually, ah, made the headlines for it”
Open-source explainable AI tools like LIME and SHAP fail enterprise scale
“If you look at the open source components they're not very scalable. They're not easy to deploy at scale. They're really, they're useful, like, if you're a data scientist, and you have your Jupyter notebook, and you're, you know, running an experiment locally,…”
Real-world AI deployments introduce distinct failure modes beyond lab environments
“And it's not only hard to develop in the lab, but once you develop it and put it in the real world, there's a whole new set of categories of ways it can go wrong.”
Historical anti-bias regulations apply directly to AI models
“There's a lot of historical regulation around anti-discrimination and bias and things like that that, ah, certainly applies just as much to AI models as it does to humans and more simple analytical models.”
Lack of trust delays enterprise AI deployments for months
“We, you know, encounter this all the time, where in organizations, they have these big plans for AI but they're just, They're unsure about actually deploying them and turning them on, and things get held up for months and months and months because of that.”