Husain: Appoint a single domain expert for open coding, not committees
Hamel Husain · Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar · Sep 25, 2025 · at 25:44
AI consultant Hamel Husain describes the 'benevolent dictator' model for conducting fast, high-signal error analysis on AI outputs.
0:00 / 0:39exact quote · 39.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Benevolent dictator is just a catchy term for the fact that when you're doing this open coding, a lot of teens get bogged down in having a committee do this. And for a lot of situations, that's wholly unnecessary. Like, You know, people get really uncomfortable with, okay, you know, we want everybody on board, we want everybody involved, so on and so forth. You need to cut through the noise. In a lot of organizations, if you look really deeply, especially small, medium-sized companies, there's really, like, you can appoint one person whose tastes that you trust.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Hamel Husain
Husain: Coding agents differ from other AI products because devs dogfood them
“Coding agents are fundamentally very different than other AI products because the developer is the domain expert. So you can short circuit a lot of things, and also the developer is using it all day long.”
Husain: General LLM benchmarks do not correlate with product-specific evals
“Up until now, a lot of the big labs understandably focused on general benchmarks, like MMLU score, human eval, things like that, which are very important for foundation models. And, you know, those not very related to product specific evals, like the ones we t…”
Husain: Product managers, not developers, must lead AI trace error analysis
“Product people have to be in the room and they have to be involved in sort of doing this. You know, usually a developer is not suited to do this, especially if it's not a coding application.”
Husain: LLM-as-a-judge evaluators must use binary scores instead of 1-5 scales
“When you go to building an LLM as a judge, you need a binary score. You don't want to think about, is this like a one, two, three, four, five, like assign a score to it. You can't, that's going to slow it down.”
Husain: Jumping straight to evals without error analysis derails AI products
“You want to usually ground yourself in your actual errors. You don't want to skip this step. And so the reason I'm kind of spending so much time on this is like, this is where people get lost. They go straight into evals. Like, let me just write some tests. An…”
Husain: Prioritize code-based evals over LLM judges to save cost and complexity
“So there's different kinds of evals. One is code-based, which you should try to do if you can, because they're cheaper. You don't have to, you know, LLM as a judge is something, it's like a meta eval. You have to eval that eval to make sure the LLM that's judg…”