MMLU-Pro, every mention
3 scenes · ← back to MMLU-Pro
tap a year for its mentions
every year anyone Philip Kiely 1Barak Lenz 1
Verbatim, from the transcripts: the passages where MMLU-Pro comes up
Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten
- ▶ 19:22 Philip Kiely It's not, you know, it got to a 56% on MMLU Pro, I think, so not, not quite Frontier, but if you're running this model, you haven't suffered any loss on your GLM-Five-II quality.
Building Jamba 3B: the tiny Hybrid Transformer State Space Reasoning Model - Barak Lenz, CTO of AI21
- ▶ 16:08 Barak Lenz Obviously not all tasks respond to these kinds of models in the sense that let's, let's just say tasks like MMLU ProR, which is like A, B, C, D, you'll get random.
The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka
- ▶ 1:24:46 unnamed speaker There's MMU, or there's MMU Hard, which, uh, someone recently released. 3 times in the scene