Husain: Log only the single most upstream error per trace
Hamel Husain · Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar · Sep 25, 2025 · at 22:14
AI eval instructor Hamel Husain outlines the core heuristic for fast and effective manual trace error analysis.
0:00 / 0:10exact quote · 10.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Just write down the first thing that you see that's wrong. The most upstream error. Don't worry about all the errors. Just capture the most, the first thing that you see that's wrong and stop and move on.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Hamel Husain
Husain: Coding agents differ from other AI products because devs dogfood them
“Coding agents are fundamentally very different than other AI products because the developer is the domain expert. So you can short circuit a lot of things, and also the developer is using it all day long.”
Husain: General LLM benchmarks do not correlate with product-specific evals
“Up until now, a lot of the big labs understandably focused on general benchmarks, like MMLU score, human eval, things like that, which are very important for foundation models. And, you know, those not very related to product specific evals, like the ones we t…”
Husain: Product managers, not developers, must lead AI trace error analysis
“Product people have to be in the room and they have to be involved in sort of doing this. You know, usually a developer is not suited to do this, especially if it's not a coding application.”
Husain: LLM-as-a-judge evaluators must use binary scores instead of 1-5 scales
“When you go to building an LLM as a judge, you need a binary score. You don't want to think about, is this like a one, two, three, four, five, like assign a score to it. You can't, that's going to slow it down.”
Husain: Jumping straight to evals without error analysis derails AI products
“You want to usually ground yourself in your actual errors. You don't want to skip this step. And so the reason I'm kind of spending so much time on this is like, this is where people get lost. They go straight into evals. Like, let me just write some tests. An…”
Husain: Prioritize code-based evals over LLM judges to save cost and complexity
“So there's different kinds of evals. One is code-based, which you should try to do if you can, because they're cheaper. You don't have to, you know, LLM as a judge is something, it's like a meta eval. You have to eval that eval to make sure the LLM that's judg…”