Hamel Husain
Founder & AI Consultant, Parlance Labs. On 2 shows, 2 appearances. The Shows tab opens the full record on each.
engineerfounderacademicauthor@HamelHusain ↗LinkedIn ↗hamel.dev ↗
Hamel Husain is an AI consultant and educator known for pioneering practical evaluation and error-analysis frameworks for LLM products. Previously a Staff ML Engineer at GitHub and a data scientist at Airbnb, he now co-teaches a widely attended AI evals course with Shreya Shankar.
Everything Hamel Husain said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Husain: Coding agents differ from other AI products because devs dogfood them
“Coding agents are fundamentally very different than other AI products because the developer is the domain expert. So you can short circuit a lot of things, and also the developer is using it all day long.”
Husain: General LLM benchmarks do not correlate with product-specific evals
“Up until now, a lot of the big labs understandably focused on general benchmarks, like MMLU score, human eval, things like that, which are very important for foundation models. And, you know, those not very related to product specific evals, like the ones we t…”
Husain: 80% of LLM-as-a-judge implementations are unhelpful
“I feel like a 75% LMS judge because it's low effort is kind of easy, but I would say out of the 75%, 80% is not helpful.”
Husain: General benchmarks for LLM judges provide very little value
“I put very little value in benchmarks, like general benchmarks. It's has some value, but you know, what you really need to do is like measure it in your domain and see if that alum as a judge is more aligned than like an off the shelf LLM. And what I've found …”
Husain: Product managers, not developers, must lead AI trace error analysis
“Product people have to be in the room and they have to be involved in sort of doing this. You know, usually a developer is not suited to do this, especially if it's not a coding application.”
Husain: LLM-as-a-judge evaluators must use binary scores instead of 1-5 scales
“When you go to building an LLM as a judge, you need a binary score. You don't want to think about, is this like a one, two, three, four, five, like assign a score to it. You can't, that's going to slow it down.”
Husain: Jumping straight to evals without error analysis derails AI products
“You want to usually ground yourself in your actual errors. You don't want to skip this step. And so the reason I'm kind of spending so much time on this is like, this is where people get lost. They go straight into evals. Like, let me just write some tests. An…”
Husain: Prioritize code-based evals over LLM judges to save cost and complexity
“So there's different kinds of evals. One is code-based, which you should try to do if you can, because they're cheaper. You don't have to, you know, LLM as a judge is something, it's like a meta eval. You have to eval that eval to make sure the LLM that's judg…”
Husain: Buying off-the-shelf automated AI eval tools does not work
“The top one is, hey, I can just buy a tool, plug it in, and it'll do the eval for you. Why do I have to worry about this? We live in the age of AI. Can't the AI just eval it? That's the most common misconception. And people want that so much that people do sel…”
Husain: Log only the single most upstream error per trace
“Just write down the first thing that you see that's wrong. The most upstream error. Don't worry about all the errors. Just capture the most, the first thing that you see that's wrong and stop and move on.”
Husain: Appoint a single domain expert for open coding, not committees
“Benevolent dictator is just a catchy term for the fact that when you're doing this open coding, a lot of teens get bogged down in having a committee do this. And for a lot of situations, that's wholly unnecessary. Like, You know, people get really uncomfortabl…”
Husain: Prompting LLMs with 'Axial Codes' Shortcuts Error Categorization
“LLMs know what open codes are, and they know what axial codes are, because it is a concept that's been around for a really long time. So those words help me shortcut, like, what I'm trying to do.”
Husain: Raw human-judge agreement is a misleading metric for AI evals
“Now, one thing you should know as a product manager is a lot of people go straight to this, like, agreement. They say, okay, my judge agrees with the human at some percentage of the time. Now that sounds appealing, but it's a very dangerous metric to use becau…”
Husain: AI evals are just standard data science applied to AI products
“People say the word eval is trying to kind of like carve out this new thing, and saying, you know, evals, and then A-B testing, but if you zoom out, it's the same data science as before, and I think that's what's causing the confusion is, hey, we need data sci…”
Husain: Inspecting raw trace data is the highest-ROI activity for AI builders
“Make it as easy as possible, because again, it's the most powerful activity that you can engage in. It's the highest ROI activity you can engage in.”
Husain: AI builders consistently get stuck moving demos to production
“Anytime that I try to help someone build an AI application, they always get stuck on how to move beyond a demo product. And they get stuck like how to systematically improve things and measure it.”
Husain: AI evaluation principles are evergreen unless AGI arrives
“And I found that, like, the subject is pretty evergreen, because we're not, you know, over the last year and a half, like, the same principles apply. And, you know, we're not really talking about Like, you know, using specific tools and APIs is more of a gener…”
Husain: LLM judges must be validated against domain experts
“One really huge thing about LLM as a judge, people love LLM as a judge, but you really have to make sure that you can trust the LLM as a judge. And so how do you trust and how do you trust anything is that you have to check it and you have to measure how good …”
Husain: Custom annotation web apps yield massive ROI for AI evals
“One counterintuitive thing that has an extreme value That people kind of discover maybe accidentally are, you know, if they're working with us, they discover very fast is Is this, there's a really, so you really want to look at your data a lot, and there's a r…”
Husain: Basic spreadsheet skills are enough to run AI evals
“The foundation of evals is error analysis. So like looking at your data and doing data analysis on your traces. So a lot of people, when we say data literacy, that can mean, that can sound scary, but it can come from a lot of different places. It can be, you c…”
Husain: Teams rarely associate AI underperformance with a lack of evals
“Cause like one thing that I wrestle with is like evals is a solution, but the problem is, okay, your AI doesn't work, or it doesn't work as well as you want it to. And people don't associate the solution with the problem cleanly enough. Cause they don't know. …”
The other half of the tape: Hamel Husain's own voice is left out of every number here. Other people bring the name up 10 times in 8 episodes across the shows. every mention, with the transcript →
Who brings them up most Kwindla Hultman Kramer 2Nina Lopatina 1Jo Kristian Bergum 1Eugene Yan 1Dylan Davis 1Ankur Goyal 1
Every mention by year
tap a year for its mentions
Latent Space 10
2025 5 mentions in 4 episodes 1 per episode
- Voice AI Masterclass — Kwindla Hultman Kramer and swyx
- [State of Context Engineering] Agentic RAG, Context Rot, MCP, Subagents — Nina Lopatina, Contextual
- ⚡️Anthropic vs Cognition on Multi-Agents: A Breakdown with Dylan Davis
- The Rise and Fall of the Vector DB category: Jo Kristian Bergum (ex-Chief Scientist, Vespa)
- every mention in 2025, scene by scene →
2024 5 mentions in 4 episodes 1 per episode
- Production AI Engineering starts with Evals
- [Paper Club] Who Validates the Validators? Aligning LLM-Judges with Humans (w/ Eugene Yan)
- The Winds of AI Winter (Q2 Four Wars of the AI Stack Recap)
- High Agency Pydantic over VC Backed Frameworks — with Jason Liu of Instructor
- every mention in 2024, scene by scene →
One line per show, most statements first. The link opens Hamel's full record on that show: the calibration, argument clarity, speaking style and every statement made there.
| Show | Role there | Eps | Statements | Record | |
|---|---|---|---|---|---|
 LENNY'S PODCASTLEDGER | Founder & AI Consultant, Parlance Labs | 1 | 13 | — | full record on Lenny's Podcast → |
| LATENT SPACELEDGER | Founder & AI Consultant, Parlance Labs | 1 | 9 | — | full record on Latent Space → |