StarZero All shows Try the podcast clipper →

Hamel Husain

Founder & AI Consultant, Parlance Labs · 1 appearance on the record.

computed by AI from the episodes · how this works → · full disclaimer →

engineerfounderacademicauthor@HamelHusain ↗LinkedIn ↗hamel.dev ↗

Hamel Husain is an AI consultant and educator known for pioneering practical evaluation and error-analysis frameworks for LLM products. Previously a Staff ML Engineer at GitHub and a data scientist at Airbnb, he now co-teaches a widely attended AI evals course with Shreya Shankar.

9statements → 1claims → 0claims resolved → 3.78/5average certainty → 2.22/5average debate potential → 10said about them ↓

1 not checkable as stated how the 1 claim stands · each chip opens the sources

1 prediction · 1 opinion · 7 insights · every statement was checked. The prediction and assertions are the 1 claim: statements the public record can support or contradict. 0 are resolved, and 1 names no date, number or outcome precise enough to check. Everything else (opinions, insights, what ifs, disclosures) can never be settled by the record, so it carries no assessment.

The record, in short

What the tape says about how Hamel argues and how the claims held up. Everything they said, and everything said about them, is in the tabs below.

How they sound: not measured why? →

We measure speaking style by listening to the audio itself, and a fair number needs at least 2,000 words from one person on tape we have measured. There is too little of Hamel Husain on measured tape to publish a rate. This says nothing about how they speak.

Everything Hamel Husain said on Latent Space that made the record, most notable first. Filter by type, assessment or year in the ledger →

Opinion
Husain: 80% of LLM-as-a-judge implementations are unhelpful
“I feel like a 75% LMS judge because it's low effort is kind of easy, but I would say out of the 75%, 80% is not helpful.”
Hamel Husain Mar 13, 2025 ▶ 14:58 [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar
Insight
Husain: General benchmarks for LLM judges provide very little value
“I put very little value in benchmarks, like general benchmarks. It's has some value, but you know, what you really need to do is like measure it in your domain and see if that alum as a judge is more aligned than like an off the shelf LLM. And what I've found …”
Hamel Husain Mar 13, 2025 ▶ 16:05 [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar
Insight
Husain: AI builders consistently get stuck moving demos to production
“Anytime that I try to help someone build an AI application, they always get stuck on how to move beyond a demo product. And they get stuck like how to systematically improve things and measure it.”
Hamel Husain Mar 13, 2025 ▶ 1:30 [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar
Insight
Husain: AI evaluation principles are evergreen unless AGI arrives
“And I found that, like, the subject is pretty evergreen, because we're not, you know, over the last year and a half, like, the same principles apply. And, you know, we're not really talking about Like, you know, using specific tools and APIs is more of a gener…”
Hamel Husain Mar 13, 2025 ▶ 10:32 [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar
Insight
Husain: LLM judges must be validated against domain experts
“One really huge thing about LLM as a judge, people love LLM as a judge, but you really have to make sure that you can trust the LLM as a judge. And so how do you trust and how do you trust anything is that you have to check it and you have to measure how good …”
Hamel Husain Mar 13, 2025 ▶ 12:34 [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar
Insight
Husain: Custom annotation web apps yield massive ROI for AI evals
“One counterintuitive thing that has an extreme value That people kind of discover maybe accidentally are, you know, if they're working with us, they discover very fast is Is this, there's a really, so you really want to look at your data a lot, and there's a r…”
Hamel Husain Mar 13, 2025 ▶ 17:31 [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar
Insight
Husain: Basic spreadsheet skills are enough to run AI evals
“The foundation of evals is error analysis. So like looking at your data and doing data analysis on your traces. So a lot of people, when we say data literacy, that can mean, that can sound scary, but it can come from a lot of different places. It can be, you c…”
Hamel Husain Mar 13, 2025 ▶ 21:37 [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar
Insight
Husain: Teams rarely associate AI underperformance with a lack of evals
“Cause like one thing that I wrestle with is like evals is a solution, but the problem is, okay, your AI doesn't work, or it doesn't work as well as you want it to. And people don't associate the solution with the problem cleanly enough. Cause they don't know. …”
Hamel Husain Mar 13, 2025 ▶ 23:49 [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar
Prediction Not checkable as stated
Husain: HCI and workflow evaluations will enter AI tools by 2027
“If I were to fast forward one or two years, I would expect to see those and all the tools. It just hasn't arrived yet.”
Hamel Husain Mar 13, 2025 ▶ 26:15 [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar

The other half of the tape: Hamel Husain's own voice is left out of every number here. Other people bring the name up 10 times in 8 episodes on Latent Space. every mention, with the transcript →

Who brings them up most Kwindla Hultman Kramer 2Nina Lopatina 1Jo Kristian Bergum 1Eugene Yan 1Dylan Davis 1Ankur Goyal 1

Every mention by year

tap a year for its mentions
00325420242025episodesmentions
02420242025episodes it came up in
000.821.5420242025episodesmentions per episode
2025 5 mentions in 4 episodes 1 per episode
2024 5 mentions in 4 episodes 1 per episode

Appearances (1)

EpisodeDateSpeaking time
[Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shank Mar 13, 2025 10m
Made with StarZero

Turn any episode into a week of clips.

This entire site, over 200 episodes transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Try the podcast clipper Learn more