Jun 4, 2026 · 49m · mad
OpenAI's Dan Roberts: Why AI Can Now Make Discoveries
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The MAD Podcast, host Matt Turck interviews Dan Roberts, theoretical physicist and Foundations of Reinforcement Learning lead at OpenAI, to explore how reinforcement learning and test-time compute are empowering AI models to make scientific discoveries. Dan shares insights on mathematical problem-solving, scaling laws, physics-inspired theoretical frameworks, and the evolving role of AI as an autonomous research collaborator.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 18.4% of the talking time here. How this is scored →
Matt as informed peer 3.8 Guest teaching 5.2 Guest disagreement 1.8 Matt pushing back 1.4
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 38:32 Dan rejects grokking and emergent scale narratives
Dan forcefully rejects the common industry view that AI capabilities abruptly 'grok' or emerge out of nowhere at scale, asserting that such claims reflect a failure to properly understand the underlying scaling sequence.
Hardest push from Matt ▶ 27:30 Matt challenges RL token efficiency with Karpathy's quoteMatt directly challenges Dan on the core efficiency of RL training by citing Andrej Karpathy's critique that RL sucks supervision through a straw with less than one bit of information per 10,000 tokens.
Biggest teaching moment ▶ 12:35 Dan breaks down RL vs Supervised Learning with Mario analogyDan provides a clear, insightful conceptual framework comparing passive observation of a video game to active trial-and-error play to illustrate the fundamental mechanics of RL.
Matt holds his own ▶ 28:47 Matt frames debate citing Rich Sutton on the Dwarkesh podcastMatt demonstrates deep industry knowledge by directly citing Rich Sutton's recent interview on Dwarkesh Patel's podcast regarding pure RL versus hybrid LLM+RL models.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Understanding the Foundations of Reinforcement Learning Team | 1 | 4 | 1 | 0 | Matt asks open-ended introductory questions about Dan's role at OpenAI and his academic trajectory. Dan details his work on RL foundations, scaling laws, and his theoretical physics background studying quantum chaos and black hole information processing. | |
| The Evolution of AI in Autonomous Scientific Discovery | 4 | 5 | 1 | 1 | Matt demonstrates domain knowledge by referencing recent developments on Erdos problems by OpenAI, DeepMind, and Anthropic. Dan educates Matt on how OpenAI refuted a lower-bound conjecture via algebraic number theory and contrasts DeepMind's Lean auto-formalization with OpenAI's natural language proof generation. | |
| Defining Reinforcement Learning: Analogies and Core Principles | 2 | 6 | 1 | 0 | Matt asks fundamental definition questions about RL, why it works, and how it fails. Dan delivers an educational breakdown using a Super Mario Bros analogy to contrast supervised learning from demonstrations with interactive reinforcement learning. | |
| Applying RL to Large Language Models and Proxy Reward Models | 4 | 5 | 1 | 1 | Matt prompts Dan on RLHF history, reward proxies, AlphaGo's Move 37, and exploration vs exploitation. Dan shares a story about Noam Brown's poker bot strategy to explain equilibrium play versus game-theory exploitation. | |
| Reframing the Paradigm: Why RL is the Main 'Cake' | 6 | 5 | 3 | 4 | Matt actively pushes back by citing Yann LeCun's 'cake' meme and Andrej Karpathy's quote about RL 'sucking supervision through a straw.' Dan defends the paradigm by emphasizing that test-time compute and reasoning breakthroughs justify the compute investment. | |
| Language as the Core Medium of Intelligence: Debating Rich Sutton | 5 | 6 | 4 | 2 | Matt references Rich Sutton's argument on the Dwarkesh podcast asserting that LLMs are not true intelligence. Dan strongly disagrees with Sutton's premise, arguing language is the essential medium of intelligence and challenging the simple 'Bitter Lesson' view of raw scaling. | |
| Demystifying Test-Time Compute and Chain of Thought | 3 | 5 | 1 | 1 | Matt asks about the internal mechanics of test-time compute and chain of thought. Dan demystifies the process, explaining that generated tokens function as a scratchpad enabling larger total compute per response. | |
| Defining Verifiable Rewards in AI | 5 | 6 | 3 | 2 | Matt asks how RL can apply to non-verifiable domains and how theoretical physics applies to AI. Dan refutes the narrative of abrupt emergent behavior or grokking, arguing scaling sequences can be made smooth using physics-style simplified modeling. | |
| The Search for a Thermodynamics of AI | 5 | 6 | 2 | 2 | Matt poses a high-level question about creating a 'thermodynamics of AI' and brings up Dan's past joke predicting 9 years to Einstein-level AI. Dan explains scaling laws as macroscopic effective theories akin to thermodynamics. | |
| Automating AI Research and Unlocking Fundamental Science | 3 | 4 | 1 | 1 | Matt asks about timelines for self-automating AI research. Dan provides a balanced perspective, expressing optimistic excitement about using AI models to solve fundamental open questions in physics and mathematics. |
Statements from this episode (17)
OpenAI studied reinforcement learning reasoning models internally long before releasing o1
“So before we released a one and thinking reasoning models, we were studying this internally”
OpenAI's RL team researches models two generations ahead of current releases
“And to do that, we need to make thinking models and some, somewhere along the way, we interact with that process. Usually at the earlier stage for models, you know, not the next model, but things that are like the next model or the next, next model.”
AI systems evolving into fully fledged scientists will be a gradual transition
“The, there's no sharp point, or I don't think there will be a sharp point where we'll say that systems didn't, weren't able to be useful for scientific, the scientific process to their fully fledged scientists. There'll be sort of a gradual shift.”
ChatGPT disproved an Erdős conjecture using cross-disciplinary mathematical reasoning
“The big result was that this conjecture of this lower bound for the number of pairs that you can make is, is false. Not only is it false, it was false due to a really interesting connection to another field of mathematics.”
OpenAI publishes most math research results using informal language settings
“Most of our results that we publicize, as far as I can think, are all in the informal setting.”
OpenAI plans to increasingly rely on reinforcement learning to scale intelligence
“When you have a lot of compute, you want to turn that compute into intelligence in a way that's useful, and RL is one way of doing it, and we just started doing it then, and we're going to do a lot more of it now.”
Roberts: Powerful pre-trained models are necessary for effective RL and reasoning
“If you have a powerful enough pre-trained model, then it can start to do well at RL. It can start to like think at use test time compute to for instance, solve, solve math problems that it wouldn't otherwise be able to do.”
Pre-training models on language before reinforcement learning is the correct architecture
“Having the model have a prior of language and being able to like, think in language and then train on top of that, that seems like clearly the right. The right thing to do.”
Dan Roberts: AI scaling requires novel algorithms, not just pure compute
“It's not that scale is all you need. You need to also have good ideas to guide the scaling.”
Combining reinforcement learning with pre-training outperforms scaling pre-training alone
“If you were just trying to scale pre-training, you wouldn't get anywhere near as far as also trying to scale RL on top of pre-training, which is what we do now.”
Roberts: AI models improve performance by generating running thought tokens in language
“The natural way it thinks is in language. It's a language model, and so that's sort of this key insight that, that you can cause it to do better just by producing a thought process in, in, in token space, in, in language.”
OpenAI will release reinforcement learning products for consulting, banking, and legal
“I definitely think OpenAI will have amazing products that will be relevant in those domains, and some amount of RL will play a role in there.”
Dan Roberts: AI scaling laws should be analyzed big-to-small
“The way to think about scaling and say, scaling laws is not small to big, but big to small.”
Dan Roberts: AI models do not experience discontinuous emergence or grokking
“You have these crazy, huge systems that have all sorts of interesting phenomena, and, you know, if you think about it the right way, they don't grok. There's just this nice continuity.”
Continuous AI improvements render multi-year autonomous agent runs highly inefficient
“In, in general, we're not just going to like set up a system and let it think autonomously for eight years, if anything, because like the system's eight years. After will be so much more powerful that it probably doesn't make sense to let a system think for a …”
Current AI models lack research taste and problem formulation ability
“There's part of the scientific process. I think that the models haven't been imbued with yet. And I'm sure people are thinking about how to do that. You know, like what, trying to get to what is the right question as opposed to here's a well-defined thing and …”