Husain: Teams rarely associate AI underperformance with a lack of evals
Hamel Husain · [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar · Mar 13, 2025 · at 23:49
AI consultant and educator Hamel Husain discusses why developers and companies struggle with AI evaluation adoption during a conversation with Latent Space podcast host Swix.
0:00 / 0:15exact quote · 15.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Cause like one thing that I wrestle with is like evals is a solution, but the problem is, okay, your AI doesn't work, or it doesn't work as well as you want it to. And people don't associate the solution with the problem cleanly enough. Cause they don't know. They don't know what they don't know.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Hamel Husain
Husain: 80% of LLM-as-a-judge implementations are unhelpful
“I feel like a 75% LMS judge because it's low effort is kind of easy, but I would say out of the 75%, 80% is not helpful.”
Husain: General benchmarks for LLM judges provide very little value
“I put very little value in benchmarks, like general benchmarks. It's has some value, but you know, what you really need to do is like measure it in your domain and see if that alum as a judge is more aligned than like an off the shelf LLM. And what I've found …”
Husain: AI builders consistently get stuck moving demos to production
“Anytime that I try to help someone build an AI application, they always get stuck on how to move beyond a demo product. And they get stuck like how to systematically improve things and measure it.”
Husain: AI evaluation principles are evergreen unless AGI arrives
“And I found that, like, the subject is pretty evergreen, because we're not, you know, over the last year and a half, like, the same principles apply. And, you know, we're not really talking about Like, you know, using specific tools and APIs is more of a gener…”
Husain: LLM judges must be validated against domain experts
“One really huge thing about LLM as a judge, people love LLM as a judge, but you really have to make sure that you can trust the LLM as a judge. And so how do you trust and how do you trust anything is that you have to check it and you have to measure how good …”
Husain: Custom annotation web apps yield massive ROI for AI evals
“One counterintuitive thing that has an extreme value That people kind of discover maybe accidentally are, you know, if they're working with us, they discover very fast is Is this, there's a really, so you really want to look at your data a lot, and there's a r…”