Husain: Coding agents differ from other AI products because devs dogfood them
Hamel Husain · Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar · Sep 25, 2025 · at 1:13:53
Hamel Husain explains why evaluation workflows that work for coding tools like Claude Code cannot be generalized to AI products in other specialized domains.
0:00 / 0:16exact quote · 16.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Coding agents are fundamentally very different than other AI products because the developer is the domain expert. So you can short circuit a lot of things, and also the developer is using it all day long.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Hamel Husain
Husain: General LLM benchmarks do not correlate with product-specific evals
“Up until now, a lot of the big labs understandably focused on general benchmarks, like MMLU score, human eval, things like that, which are very important for foundation models. And, you know, those not very related to product specific evals, like the ones we t…”
Husain: Product managers, not developers, must lead AI trace error analysis
“Product people have to be in the room and they have to be involved in sort of doing this. You know, usually a developer is not suited to do this, especially if it's not a coding application.”
Husain: LLM-as-a-judge evaluators must use binary scores instead of 1-5 scales
“When you go to building an LLM as a judge, you need a binary score. You don't want to think about, is this like a one, two, three, four, five, like assign a score to it. You can't, that's going to slow it down.”
Husain: Jumping straight to evals without error analysis derails AI products
“You want to usually ground yourself in your actual errors. You don't want to skip this step. And so the reason I'm kind of spending so much time on this is like, this is where people get lost. They go straight into evals. Like, let me just write some tests. An…”
Husain: Prioritize code-based evals over LLM judges to save cost and complexity
“So there's different kinds of evals. One is code-based, which you should try to do if you can, because they're cheaper. You don't have to, you know, LLM as a judge is something, it's like a meta eval. You have to eval that eval to make sure the LLM that's judg…”
Husain: Buying off-the-shelf automated AI eval tools does not work
“The top one is, hey, I can just buy a tool, plug it in, and it'll do the eval for you. Why do I have to worry about this? We live in the age of AI. Can't the AI just eval it? That's the most common misconception. And people want that so much that people do sel…”