MMLU
product on 7 shows · 11 statements across 11 episodes · said 40 times in 24 episodes since 2024
Latent Space 22 
No Priors 5 
the a16z Podcast 3 
Big Technology 2 
TBPN 2 
the Y Combinator Startup Podcast 1 
Lenny's Podcast
Mentions by year, every show
tap a year for its mentions
Latent Space 22No Priors 5the a16z Podcast 3Big Technology 2TBPN 2the Y Combinator Startup Podcast 1
2026 3 mentions in 1 episode
2025 13 mentions in 10 episodes 1 per episode
- Beyond Leaderboards: LMArena’s Mission to Make AI Reliable
- No Priors Ep. 105 | With Director of the Center of AI Safety Dan Hendrycks
- 2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents
- Anthropic Head of Pretraining on Scaling Laws, Compute, and the Future of AI · Y Combinator
- Better Data is All You Need — Ari Morcos, Datology
- Weekly Recap: Apple Sitting on $100B, Disney Enters The AI Race, Marc Andreessen, All About GPT-5
- The State Of AI in 15 Minutes
- ⚡️Multi-Turn RL for Multi-Hour Agents — with Will Brown, Prime Intellect
- 2 more episodes that year, every mention in 2025 →
2024 19 mentions in 10 episodes 2 per episode
- No Priors Ep. 77 | With Foundry CEO and Founder Jared Quincy Davis
- Training Llama 2, 3 & 4: The Path to Open Source AGI — with Thomas Scialom of Meta AI
- The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka
- [Paper Club] Upcycling Large Language Models into Mixture of Experts
- AI Scaling, Alignment, and the Path to Superintelligence — With Dwarkesh Patel
- A Comprehensive Overview of Large Language Models - Latent Space Paper Club
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- [Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz
- 2 more episodes that year, every mention in 2024 →
every mention on every show, scene by scene, with the transcript →
11 statements about MMLU, every show
Hill-Smith: Google used unpublished 32-shot CoT to claim Gemini beat GPT-4
“Back when I'm Googled a Gemini one when I ultra and needed a number that would say it was better than GPT four. And Like, constructed I think never published, like, chain of thought examples, 32 of them in every topic in MLU to run it, to get the score.”
Bakouch: SmolLM 2 scored random on MMLU until 6.5 trillion tokens
“And this is basically until like 6.5 trillion of tokens, which is a lot, to be honest. Until this amount of token, the MMLU in the QA format, meaning that the model have to select which answer is, the model have to output, for example, the right answer is A, o…”
Husain: General LLM benchmarks do not correlate with product-specific evals
“Up until now, a lot of the big labs understandably focused on general benchmarks, like MMLU score, human eval, things like that, which are very important for foundation models. And, you know, those not very related to product specific evals, like the ones we t…”
Midha: Real-time testing will replace static AI benchmarks like MMLU
“While benchmarks like MMLU and the idea of these static exams were useful three years ago, the future is about real-time evaluation, real-time systems, real-time testing in the wild.”
Beauchamp: Real AI progress is performance per dollar, not raw benchmarks
“For us, it doesn't make sense to think of AI as just the absolute performance. So if you look at like the MMLU score or the, you know, any of these benchmarks that people like to look at, If you just get that score, it doesn't really tell, tell you anything. C…”
Synthetic college text boosts MMLU; middle school text boosts OpenBookQA
“College textbooks are really good for MLU or middle school textbooks are good for benchmarks like open book, UA and Pico.”
He: Upcycling a 15B model on 1T tokens yielded 4% MMLU gain
“On other scaling experiments, we tried on 15 B models upcycling and applied on one trillion tokens and achieved roughly about five percent improvement in terms of the validation loss and four percent improvement on MMLU.”
Schulhoff: Role Prompting Does Not Improve Accuracy on Modern LLMs
“For accuracy-based tasks, like MMLU, you're trying to solve a math problem, and maybe you tell the AI that it's a math professor, and you expect it to have improved performance. I really don't think that works. I'm quite certain that doesn't work on more moder…”
Davis: Compound AI approach yielded a 3% MMLU performance bump
“The MMLU performance bump was about three percent. And to put that in perspective, the gap between some of the previous best models is often less than one percent. Between, for example, Gemini, 1.5, and Lama 3.1, and things like that. So, actually, 2.8% or thr…”
Standard AI benchmarks like MMLU are becoming saturated and inadequate
“I mean, like, people will come out and say, here's what we got on MMLU and so on, but they're getting saturated, and they're not, often not that great to begin with, so I, I'm more eager to see what it feels like to talk to one of these things than learn what …”
Ghodsi: MMLU benchmark scores are inflated due to web data contamination
“MMLU is just a multi-choice question that's on the web. Ask a question. Here's, is the answer A, B, C, D, and then it says what the right answer is. And it's on the web. You can deliberately train on it and create an LLM that crushes it on that. Okay. Or you c…”