HumanEval
also referred to as: human eval
10 statements across 9 episodes · 2 bullish · 7 bearish · 9 people on the record · first statement Nov 3, 2023 by Michael Royzen · said 39 times in 15 episodes since 2023 · across every show →
Mentions by year
brought up most by Varun Mohan (5), Anshul Ramachandran (5), Alessio Fanelli (5), Shawn Wang (3), Shunyu Yao (2), Michael Royzen (2), Jesse Hu (2), Diego Bachman (2)
tap a year for its mentions
2026 2 mentions in 1 episode
2025 11 mentions in 3 episodes 4 per episode
2024 23 mentions in 9 episodes 3 per episode
- [Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- [LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models
- Windsurf: The Enterprise AI IDE
- Language Agents: From Reasoning to Acting — with Shunyu Yao of OpenAI, Harrison Chase of LangGraph
- Is finetuning GPT4o worth it?
- Breaking down the OG GPT Paper by Alec Radford
- Training Llama 2, 3 & 4: The Path to Open Source AGI — with Thomas Scialom of Meta AI
- 1 more episode that year, every mention in 2024 →
2023 3 mentions in 2 episodes 2 per episode
Everything said about HumanEval, oldest first
Nov 3, 2023 negative
Royzen: GPT-4 was trained on HumanEval, proving data contamination
“GPT-IV itself has been trained on human eval, and we know this because GPT-IV is able to predict the exact doc string in many of the problems. I've seen it predict, like, the specific example values in the doc string, which is extremely improbable for it to ju…”
Jul 5, 2024 bearish
Yi Tay: GSM8K and HumanEval are saturated, contaminated, and uninformative
“I mean, like, you know, the things like GSMK human eval, the coding human eval, they're all, like Contaminated. Like, not, not, I wouldn't say, they're all, like, saturated, contaminated, you know, like, you know, GSMK, whether you're a 92, 91, like, no one ca…”
Aug 22, 2024 negative
Training on pull request diffs yields code generators, not software engineers
“What the, very crudely, what the pre-trained models are reading is they're reading those final diffs and they're Emulating that and then being able to output it. Right. But of course it's a super lossy thing, a PR. You have no idea why or how, for the most par…”
Oct 19, 2024 bearish
Nov 28, 2024 positive
Schluntz: Traditional coding evals remain useful alongside SWE-bench
“I think there's definitely a space for these more traditional coding evals that are sort of easy to implement, quick to run and do get you some signal. And maybe hopefully there's just sort of harder versions of human eval that get created.”
Dec 13, 2024 negative
Ramachandran: SWE-bench and HumanEval do not reflect real professional engineering
“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have sweet bench. That's cool. No actual Professional work looks like Sweebench. Like, human eval, same thi…”
Jul 28, 2025 negative
Ramachandran: SWE-bench and HumanEval do not reflect real professional software engineering
“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have SweeBench, that's cool, no actual Professional work looks like Sweebench, like human eval, same thing.”
Jul 28, 2025 negative
Mohan: HumanEval benchmark scores are inflated due to GitHub training contamination
“One of the issues that ends up coming up with things like human eval is contamination, because a lot of these things that train models end up training on all of GitHub. GitHub itself has human eval. So they end up Training on that, and then the numbers are arb…”
Sep 23, 2025 positive