AI evals
3 statements across 1 episodes · 0 bullish · 2 bearish · 2 people on the record · first statement Sep 25, 2025 by Hamel Husain · across every show →
Everything said about AI evals, oldest first
Sep 25, 2025 negative
Husain: Buying off-the-shelf automated AI eval tools does not work
“The top one is, hey, I can just buy a tool, plug it in, and it'll do the eval for you. Why do I have to worry about this? We live in the age of AI. Can't the AI just eval it? That's the most common misconception. And people want that so much that people do sel…”
Sep 25, 2025 negative
Husain: Raw human-judge agreement is a misleading metric for AI evals
“Now, one thing you should know as a product manager is a lot of people go straight to this, like, agreement. They say, okay, my judge agrees with the human at some percentage of the time. Now that sounds appealing, but it's a very dangerous metric to use becau…”
Sep 25, 2025
Shankar: Establishing initial AI evals takes 3–4 days plus 30 minutes weekly
“Usually I'll spend three to four days really working with whoever to do initial rounds of error analysis, like a lot of labeling, feel like we're in a good place to create the spreadsheet that Hamill had and everyone's kind of on board and convinced, and even …”