Jan 22, 2020 · 21m · mad
Production AI: Lessons Learned the Hard Way // Adam Wenchel, Arthur.ai (FirstMark's Data Driven NYC)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
At FirstMark's Data Driven NYC, Arthur AI CEO Adam Wenchel presents 'Production AI: Lessons Learned the Hard Way,' detailing why real-world machine learning models degrade after deployment and how enterprise monitoring and explainability guardrails bridge the gap between lab performance and production reliability.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 3.9% of the talking time here. How this is scored →
Matt as informed peer 0.7 Guest teaching 2.0 Guest disagreement 0.2 Matt pushing back 0.5
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 20:50 Wenchel Playfully Critiques Academic Open Source Code
Wenchel lightheartedly dismisses open-source explainability scripts like Lime as unscalable code written by PhDs purely to support academic papers.
Hardest push from Matt ▶ 17:10 Matt Turck Challenges Data Access FeasibilityHost Matt Turck pushes Wenchel on whether Arthur requires access to raw underlying data, forcing Wenchel to clarify proxy metrics versus ground truth monitoring.
Biggest teaching moment ▶ 17:31 Wenchel Differentiates Delayed Outcomes vs Instant FeedbackWenchel educates the host and audience on how to monitor models like credit underwriting where ground truth outcomes take years to materialise.
Matt holds his own ▶ 15:20 Matt Turck Questions Model Agnosticism Across FrameworksHost Matt Turck demonstrates domain knowledge by asking whether Arthur's architecture can handle diverse paradigms like deep learning, NLP, and tabular data.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Speaker Background and Audience Engagement | 0 | 1 | 0 | 0 | Adam Wenchel opens with his background at DARPA and Capital One before introducing the topic. As this is a monologue presentation, host-side scores are zero. | |
| Bad Decisions, Model Degradation, and Operational Gaps | 0 | 2 | 0 | 0 | Wenchel details how AI models degrade in production due to changing macroeconomic conditions and upstream data format changes. The segment remains a monologue, keeping host scores at zero. | |
| Inappropriate Decisions, Bias, and Regulatory Compliance | 0 | 2 | 0 | 0 | Wenchel addresses issues of regulatory compliance, bias monitoring, and trust in deployed models. Host involvement is zero during this presentation section. | |
| Case Study: Dumbarton Oaks Harvard Research | 0 | 2 | 0 | 0 | Wenchel presents a Harvard Dumbarton Oaks case study showing how computer vision explainability identified vegetation misclassifications in nave architecture. Host scores remain zero. | |
| Case Study: US Air Force Supply Chain Optimization | 0 | 2 | 0 | 0 | Wenchel covers the US Air Force supply chain optimization case study and concludes his deck with a hiring pitch. The segment is entirely guest-led with no host interaction. | |
| Q&A: Platform Architecture and Model Agnosticism | 4 | 3 | 1 | 3 | Host Matt Turck and audience members ask targeted Q&A questions regarding data ingestion, benchmark comparison, model agnosticism, and dataset access. Turck demonstrates strong technical understanding when questioning data access limits. |
Statements from this episode (12)
Real-world AI deployments introduce distinct failure modes beyond lab environments
“And it's not only hard to develop in the lab, but once you develop it and put it in the real world, there's a whole new set of categories of ways it can go wrong.”
Historical anti-bias regulations apply directly to AI models
“There's a lot of historical regulation around anti-discrimination and bias and things like that that, ah, certainly applies just as much to AI models as it does to humans and more simple analytical models.”
Lack of trust delays enterprise AI deployments for months
“We, you know, encounter this all the time, where in organizations, they have these big plans for AI but they're just, They're unsure about actually deploying them and turning them on, and things get held up for months and months and months because of that.”
Most companies using AI suffer from unpublicized model failures
“Every company that's doing anything substantive with AI probably suffers from any of these problems, it's just a few of them have actually, ah, made the headlines for it”
Deployed AI models suffer immediate performance gaps and ongoing degradation
“The second you put it in the real world, models, ah, number one, there's a gap right from day one, and they get worse over time.”
Not collecting protected class data does not prevent algorithmic bias
“What's happened a lot in the past is people have kind of like taken the head in the sand approach where they've sort of said like, oh you know, we're not even collecting protected classes, so we can't possibly be biased as far as we know, and that's no longer …”
A fractional drop in AI performance can cost hundreds of millions
“Even if your model encounters some sort of issue that drops at a couple 10th of a percent in performance, that can literally be hundreds of millions of dollars over time, and so the ROI on having that kind of monitoring in place is, is huge.”
Enterprise AI adoption fails without risk mitigation despite performance gains
“Especially large traditional enterprises there tend to be very consensus-driven cultures by nature, and so the, even if people, if a data scientist can demonstrate they have a model that, you know, generally, like, gets some huge five or 10% lift, which, you k…”
Visual AI explainability helps build trust in skeptical academic communities
“Ah, and then the other thing is, you can imagine this world of humanities research has not changed a whole lot in the, like, the last 200 years of study, and so bringing this sort of innovation, like, we can automate this and computers can find patterns that w…”
Early safety guardrails enable companies to pursue more aggressive AI strategies
“Like the more you can build these guardrails in from kind of day one of your AI projects, when you, it allows you to be more aggressive, right? Just like the safety systems on an F-one car allow you to lap faster. If you build this stuff in from the beginning …”
Credit underwriting AI feedback loops take three to four years
“There's other ones like, ah, underwriting credit cards, where you might not know for three or four years whether you should have given that person a credit card, right?”
Open-source explainable AI tools like LIME and SHAP fail enterprise scale
“If you look at the open source components they're not very scalable. They're not easy to deploy at scale. They're really, they're useful, like, if you're a data scientist, and you have your Jupyter notebook, and you're, you know, running an experiment locally,…”