Krieger: AI benchmark evals often contain flawed ground-truth answers
Mike Krieger · A conversation with OpenAI's CPO Kevin Weil, Anthropic's CPO Mike Krieger, and Sarah Guo · Nov 6, 2024 · at 16:50
Mike Krieger (CPO, Anthropic) discusses how AI product builders should develop intuition around model evaluation and why aggregate benchmark scores can be misleading.
0:00 / 0:27exact quote · 27.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And some of these model these evals we've seen, like, even the golden answer, I'm like, I'm not sure a human would say it, or, like, I think that math is actually a little wrong, like, getting a hundred percent is gonna be really hard, because even just grading them is very challenging, so, like, I'd encourage you, like, the way you build intuition is go look at the actual answers, even the sample of them, and be like, alright, yeah, maybe we should evolve the evals, or maybe, like, the vibes are good, even if the eval is, like, tight, so, like, getting real and getting, like, deep on the data, I think matters.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Mike Krieger
Mike Krieger estimates Claude Code is over 95% written by AI
“At this point, I would be shocked if it wasn't 95% plus.”
Anthropic's Claude Code team uses Claude to write and review itself
“The team that works in the most futuristic way is the Claude code team, because they're using Claude code to build Claude code in a very self-improving kind of way. And, you know, early on in that project, they would do very line by line pull request reviews, …”
Mike Krieger expects AI to autonomously resolve user feedback by 2025
“Hey, I'm in the discord, the, you know, the, Cloud anthropic discord. I'm in the user for I'm on X and I'm reading things and like, here's what's emergent. That's step one. Models can do that today. Step two, which the models probably can do today. We just hav…”
Mike Krieger says MCP is more promising for agents than computer use
“To really have agency and have these agentic use cases. Like one way you approach it is computer use, but computer use has a bunch of limitations. The way I get way more excited about everything is an MCP and our models are really good at using MCPs. All of a …”
Mike Krieger uses Claude Opus 4 as his primary strategy partner
“Where my go-to product strategy partner is Claude, and it has been basically for that full year where I'll write an initial strategy. I'll share it with Claude basically, and I'll have it, you know, look at it. And in the past, it's pretty anodyne kind of comm…”
Anthropic models hit 72% on SWE-bench, crushing earlier timeline predictions
“He's like, I think we'll be at 90% by the end of 20, 25 or something like that. And sure enough, we're at about 72 now with the new models and. We're at 50% when you made that prediction and it's like continued to scale pretty much like as predicted.”