Sep 25, 2025 · 1h 46m · lennys-podcast
Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this masterclass, AI evaluation experts Hamel Husain and Shreya Shankar demonstrate how product builders can transition from guesswork and 'vibe checks' to systematic, high-ROI error analysis and automated evaluators.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 21.7% of the talking time here. How this is scored →
Lenny as informed peer 3.9 Guest teaching 4.8 Guest disagreement 1.0 Lenny pushing back 0.4
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:13:13 Debunking the 'No Evals, Just Vibes' Coding Agent Myth
Shreya and Hamel push back strongly against the viral narrative that top products like Claude Code operate solely on vibe checks without rigorous foundational evals and telemetry.
Hardest push from Lenny ▶ 8:29 Lenny Testing the Unit Test MetaphorLenny presses on whether evals can simply be thought of as standard software unit tests, challenging the guests to clarify the exact functional distinction.
Biggest teaching moment ▶ 58:20 Exposing the Trap of High Percentage Agreement in EvalsHamel demonstrates why a product manager celebrating a 90% judge agreement metric might actually be masking massive failure rates on rare edge cases without a confusion matrix.
Lenny holds their own ▶ 1:00:56 Connecting Automated Judges Directly to Product Requirements DocumentsLenny synthesizes the guests' complex prompt engineering mechanics into the core product management discipline, demonstrating how judge prompts function as continuously executing dynamic PRDs.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| Preview: Why AI Evals Are the Hottest Skill | 3 | 1 | 0 | 0 | Lenny delivers an introductory monologue summarizing the rising importance of AI evals and framing Hamel and Shreya's course on Maven before cutting to sponsor reads. | |
| Sponsor: Fin AI Agent by Intercom | 4 | 5 | 1 | 1 | Lenny proposes that evals are simply unit tests for AI, prompting Shreya to gently reframe the concept into a much wider spectrum of data analytics and qualitative discovery. | |
| Demonstrating Error Analysis and Open Coding on Real Traces | 3 | 6 | 1 | 0 | Hamel leads a detailed walkthrough of real-world conversational traces from Nurture Boss, instructing Lenny on open coding and capturing upstream error notes manually. | |
| The Limits of Automation and the Benevolent Dictator | 3 | 6 | 2 | 1 | Shreya warns against early LLM automation for note-taking due to lack of domain context, and Hamel explains the necessity of appointing a single benevolent dictator over a committee. | |
| Synthesizing Open Codes into Axial Categories | 4 | 5 | 1 | 0 | The guests explain theoretical saturation and demonstrate clustering open codes into axial categories with LLMs, referencing foundational machine learning error analysis roots from Andrew Ng. | |
| Quantifying Error Distributions and Prioritizing Fixes | 3 | 5 | 0 | 0 | Hamel demonstrates aggregating categorized errors using pivot tables to quantitatively prioritize application fixes before jumping into automated testing. | |
| Code-Based Evaluators Versus LLM-as-a-Judge | 4 | 6 | 2 | 0 | Shreya and Hamel distinguish deterministic code evaluators from LLM judges, strongly criticizing 1-to-5 Likert scales in favor of strict binary pass-fail metrics. | |
| Sponsor: Mercury Financial Services for Startups | 4 | 6 | 1 | 0 | After an ad break, Hamel details aligning LLM judges with human evaluations and warns why overall percentage agreement is misleading without an error confusion matrix. | |
| Evals as Dynamic PRDs and the Impact of Criteria Drift | 6 | 5 | 1 | 1 | Lenny makes an insightful connection between LLM judge prompts and dynamic PRDs, leading Shreya to discuss her research on criteria drift during model evaluation. | |
| Debating Vibe Checks, Coding Agents, and A/B Testing | 5 | 6 | 3 | 2 | The guests dismantle online debates pitting vibes and A/B testing against evals, explaining that coding agents benefit from unique developer-dogfooding dynamics that do not generalize. | |
| Common Misconceptions, Custom Tools, and Practical Advice | 4 | 5 | 1 | 0 | Hamel and Shreya emphasize looking directly at raw traces and show how simple custom internal interfaces maximize iteration velocity and product ROI. | |
| Overview of the Maven AI Evals Course and Resources | 4 | 4 | 0 | 0 | The guests outline their Maven curriculum, supplemental comprehensive handbook, and custom AI assistant bot trained on their entire corpus. | |
| Lightning Round: Books, Culture, Tools, and Mottos | 3 | 3 | 0 | 0 | The interview concludes with a lighthearted lightning round touching on textbooks, fiction, TV shows, favorite developer tools, and mutual admiration between the guests. |
Statements from this episode (21)
Husain: Product managers, not developers, must lead AI trace error analysis
“Product people have to be in the room and they have to be involved in sort of doing this. You know, usually a developer is not suited to do this, especially if it's not a coding application.”
Husain: Log only the single most upstream error per trace
“Just write down the first thing that you see that's wrong. The most upstream error. Don't worry about all the errors. Just capture the most, the first thing that you see that's wrong and stop and move on.”
Shankar: LLMs fail at initial error analysis due to missing context
“What we usually find when we try to ask an LLM to do this error analysis is it just says the trace looks good because it doesn't have the context needed to understand whether something might be, you know, bad product smell or, you know, not”
Husain: Appoint a single domain expert for open coding, not committees
“Benevolent dictator is just a catchy term for the fact that when you're doing this open coding, a lot of teens get bogged down in having a committee do this. And for a lot of situations, that's wholly unnecessary. Like, You know, people get really uncomfortabl…”
Husain: LLM-as-a-judge evaluators must use binary scores instead of 1-5 scales
“When you go to building an LLM as a judge, you need a binary score. You don't want to think about, is this like a one, two, three, four, five, like assign a score to it. You can't, that's going to slow it down.”
Husain: Prompting LLMs with 'Axial Codes' Shortcuts Error Categorization
“LLMs know what open codes are, and they know what axial codes are, because it is a concept that's been around for a really long time. So those words help me shortcut, like, what I'm trying to do.”
Shankar: Open code notes must be detailed for LLMs to categorize errors
“This also drives home the point that your open codes have to be detailed, right? You can't just say janky because if the AI is reading janky, it's not going to be able to categorize it. Even a human wouldn't, right? It would have to go and remember why you sai…”
Husain: Jumping straight to evals without error analysis derails AI products
“You want to usually ground yourself in your actual errors. You don't want to skip this step. And so the reason I'm kind of spending so much time on this is like, this is where people get lost. They go straight into evals. Like, let me just write some tests. An…”
Husain: Prioritize code-based evals over LLM judges to save cost and complexity
“So there's different kinds of evals. One is code-based, which you should try to do if you can, because they're cheaper. You don't have to, you know, LLM as a judge is something, it's like a meta eval. You have to eval that eval to make sure the LLM that's judg…”
Shankar: LLM judges are reliable when scoped to binary failure modes
“People always think like, oh, this is at least as hard as my problem of creating the original agent, and it's not because you're asking the judge to do one thing, evaluate one failure mode. So the scope of the problem is very small and the output of this LLM j…”
Husain: Raw human-judge agreement is a misleading metric for AI evals
“Now, one thing you should know as a product manager is a lot of people go straight to this, like, agreement. They say, okay, my judge agrees with the human at some percentage of the time. Now that sounds appealing, but it's a very dangerous metric to use becau…”
Rachitsky: Automated eval judges are the purest form of modern PRDs
“I've had some guests on the podcast recently who've been saying evals are the new PRDs. And if you look at this is exactly what this is like. Product managers, product teams, right? Here's what the product should be. Here's all the requirements. Here's like th…”
Shreya Shankar: AI eval rubrics cannot be defined upfront without data
“What's new here is that you can't figure out your rubrics upfront. People's opinions of good and bad change as they review more outputs. They think of failure modes only after seeing 10 outputs they would never have dreamed of in the first place.”
Shreya Shankar: AI products typically need only four to seven LLM evals
“For me, like, between four and seven. It's not that many, because a lot of the failure modes, as Hamill said earlier, can be fixed by just fixing your prompt.”
Shreya Shankar: AI companies conceal evals because they are competitive moats
“And people don't talk about it because this is their moat, right? So people are not going to go and share all of these things because it makes sense, right? If you are an email writing assistant and you're doing this and you're doing it well, you don't want so…”
Husain: Coding agents differ from other AI products because devs dogfood them
“Coding agents are fundamentally very different than other AI products because the developer is the domain expert. So you can short circuit a lot of things, and also the developer is using it all day long.”
Husain: AI evals are just standard data science applied to AI products
“People say the word eval is trying to kind of like carve out this new thing, and saying, you know, evals, and then A-B testing, but if you zoom out, it's the same data science as before, and I think that's what's causing the confusion is, hey, we need data sci…”
Husain: General LLM benchmarks do not correlate with product-specific evals
“Up until now, a lot of the big labs understandably focused on general benchmarks, like MMLU score, human eval, things like that, which are very important for foundation models. And, you know, those not very related to product specific evals, like the ones we t…”
Husain: Buying off-the-shelf automated AI eval tools does not work
“The top one is, hey, I can just buy a tool, plug it in, and it'll do the eval for you. Why do I have to worry about this? We live in the age of AI. Can't the AI just eval it? That's the most common misconception. And people want that so much that people do sel…”
Husain: Inspecting raw trace data is the highest-ROI activity for AI builders
“Make it as easy as possible, because again, it's the most powerful activity that you can engage in. It's the highest ROI activity you can engage in.”
Shankar: Establishing initial AI evals takes 3–4 days plus 30 minutes weekly
“Usually I'll spend three to four days really working with whoever to do initial rounds of error analysis, like a lot of labeling, feel like we're in a good place to create the spreadsheet that Hamill had and everyone's kind of on board and convinced, and even …”