Douglas: Frontier Labs Avoid RL Training Directly on ARC-AGI
Sholto Douglas · Weekly Recap - Elon Vs Trump, Ukraine's Drone Attack, Cluely Update & OpenAI CRO · Jun 7, 2025 · at 1:55:15
Sholto Douglas, AI researcher at Anthropic, discusses benchmark evaluation and how frontier labs maintain the integrity of ARC-AGI.
0:00 / 0:10exact quote · 10.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And I mean, I think if you are old on Arc AGI, then it would, you'd probably get superhuman at it pretty fast. But I think we're all trying not to RL on it so that it functions as like an interesting held out.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Sholto Douglas
Douglas: Generalist models will obsolete specialized fine-tuned models
“I really do think that similar to how we saw with large pre-trained models before with small fine-tuned models made it like, had gains over the sort of GPT-II era, but then were obsoleted by GPT-IV being generally good at everything. I think, to be honest, you…”