OpenAI's RL team researches models two generations ahead of current releases
Dan Roberts · OpenAI's Dan Roberts: Why AI Can Now Make Discoveries · Jun 4, 2026 · at 2:54
Dan Roberts, lead of the Foundations of Reinforcement Learning team at OpenAI, describes his team's focus on long-term R&D rather than immediate model deployments.
0:00 / 0:13exact quote · 13.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And to do that, we need to make thinking models and some, somewhere along the way, we interact with that process. Usually at the earlier stage for models, you know, not the next model, but things that are like the next model or the next, next model.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dan Roberts
Dan Roberts: AI scaling requires novel algorithms, not just pure compute
“It's not that scale is all you need. You need to also have good ideas to guide the scaling.”
Dan Roberts: AI models do not experience discontinuous emergence or grokking
“You have these crazy, huge systems that have all sorts of interesting phenomena, and, you know, if you think about it the right way, they don't grok. There's just this nice continuity.”
Pre-training models on language before reinforcement learning is the correct architecture
“Having the model have a prior of language and being able to like, think in language and then train on top of that, that seems like clearly the right. The right thing to do.”
Combining reinforcement learning with pre-training outperforms scaling pre-training alone
“If you were just trying to scale pre-training, you wouldn't get anywhere near as far as also trying to scale RL on top of pre-training, which is what we do now.”
ChatGPT disproved an Erdős conjecture using cross-disciplinary mathematical reasoning
“The big result was that this conjecture of this lower bound for the number of pairs that you can make is, is false. Not only is it false, it was false due to a really interesting connection to another field of mathematics.”