HumanEval
product on 3 shows · 12 statements across 11 episodes · said 40 times in 16 episodes since 2023
Latent Space 39 
BG2 Pod 1 
Lenny's Podcast
Mentions by year, every show
tap a year for its mentions
Latent Space 39BG2 Pod 1
2026 2 mentions in 1 episode
2025 11 mentions in 3 episodes 4 per episode
2024 24 mentions in 10 episodes 2 per episode
- [Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- [LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models
- Windsurf: The Enterprise AI IDE
- Language Agents: From Reasoning to Acting — with Shunyu Yao of OpenAI, Harrison Chase of LangGraph
- Is finetuning GPT4o worth it?
- Breaking down the OG GPT Paper by Alec Radford
- Training Llama 2, 3 & 4: The Path to Open Source AGI — with Thomas Scialom of Meta AI
- 2 more episodes that year, every mention in 2024 →
2023 3 mentions in 2 episodes 2 per episode
every mention on every show, scene by scene, with the transcript →
12 statements about HumanEval, every show
Hill-Smith: Early Benchmarks Like HumanEval Are Saturated and Trivial
“Well, V one would be completely saturated right now by almost every model coming out because doing things like writing the Python functions and human evil is now pretty trivial.”
Husain: General LLM benchmarks do not correlate with product-specific evals
“Up until now, a lot of the big labs understandably focused on general benchmarks, like MMLU score, human eval, things like that, which are very important for foundation models. And, you know, those not very related to product specific evals, like the ones we t…”
Bachman: PowerCoder-3B reaches 35% HumanEval accuracy versus StarCoder's 30%
“In the end, this converges to, I believe, about 35% accuracy on human eval, whereas the star coder baseline was about 30%.”
Mohan: HumanEval benchmark scores are inflated due to GitHub training contamination
“One of the issues that ends up coming up with things like human eval is contamination, because a lot of these things that train models end up training on all of GitHub. GitHub itself has human eval. So they end up Training on that, and then the numbers are arb…”
Ramachandran: SWE-bench and HumanEval do not reflect real professional software engineering
“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have SweeBench, that's cool, no actual Professional work looks like Sweebench, like human eval, same thing.”
Ramachandran: SWE-bench and HumanEval do not reflect real professional engineering
“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have sweet bench. That's cool. No actual Professional work looks like Sweebench. Like, human eval, same thi…”
Schluntz: Traditional coding evals remain useful alongside SWE-bench
“I think there's definitely a space for these more traditional coding evals that are sort of easy to implement, quick to run and do get you some signal. And maybe hopefully there's just sort of harder versions of human eval that get created.”
Jesse Hu: Frontier models have effectively capped out HumanEval performance
“I think through multiple techniques and through the most recent models, you can actually basically cap out on the performance on human eval.”
Training on pull request diffs yields code generators, not software engineers
“What the, very crudely, what the pre-trained models are reading is they're reading those final diffs and they're Emulating that and then being able to output it. Right. But of course it's a super lossy thing, a PR. You have no idea why or how, for the most par…”
Yi Tay: GSM8K and HumanEval are saturated, contaminated, and uninformative
“I mean, like, you know, the things like GSMK human eval, the coding human eval, they're all, like Contaminated. Like, not, not, I wouldn't say, they're all, like, saturated, contaminated, you know, like, you know, GSMK, whether you're a 92, 91, like, no one ca…”
Gerstner: GPT-4o Dramatically Lowers Inference Pricing Over Benchmark Gains
“And we plotted chat GPT for Omni on this chart, and you can see how it barely improved in terms of human level, the human eval score, but it dramatically improved in terms of pricing, you know, in terms of inference pricing.”
Royzen: GPT-4 was trained on HumanEval, proving data contamination
“GPT-IV itself has been trained on human eval, and we know this because GPT-IV is able to predict the exact doc string in many of the problems. I've seen it predict, like, the specific example values in the doc string, which is extremely improbable for it to ju…”