Dan Roberts
Research Lead, OpenAI. On 1 show, 1 appearance. The Shows tab opens the full record on each.
scientistauthorfounderacademic@danintheory ↗LinkedIn ↗danintheory.com ↗
He is the co-author of the textbook The Principles of Deep Learning Theory, which applies theoretical physics to neural networks and reinforcement learning. He previously earned a PhD in physics from MIT, conducted postdoctoral research at the Institute for Advanced Study, and co-founded Diffeo, which was acquired by Salesforce.
1shows
1appearances
17statements
1resolved
1supported
0contradicted
100%fully supported
Everything Dan Roberts said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Dan Roberts: AI scaling requires novel algorithms, not just pure compute
“It's not that scale is all you need. You need to also have good ideas to guide the scaling.”
Dan Roberts: AI models do not experience discontinuous emergence or grokking
“You have these crazy, huge systems that have all sorts of interesting phenomena, and, you know, if you think about it the right way, they don't grok. There's just this nice continuity.”
Pre-training models on language before reinforcement learning is the correct architecture
“Having the model have a prior of language and being able to like, think in language and then train on top of that, that seems like clearly the right. The right thing to do.”
Combining reinforcement learning with pre-training outperforms scaling pre-training alone
“If you were just trying to scale pre-training, you wouldn't get anywhere near as far as also trying to scale RL on top of pre-training, which is what we do now.”
ChatGPT disproved an Erdős conjecture using cross-disciplinary mathematical reasoning
“The big result was that this conjecture of this lower bound for the number of pairs that you can make is, is false. Not only is it false, it was false due to a really interesting connection to another field of mathematics.”
OpenAI plans to increasingly rely on reinforcement learning to scale intelligence
“When you have a lot of compute, you want to turn that compute into intelligence in a way that's useful, and RL is one way of doing it, and we just started doing it then, and we're going to do a lot more of it now.”
Dan Roberts: AI scaling laws should be analyzed big-to-small
“The way to think about scaling and say, scaling laws is not small to big, but big to small.”
Current AI models lack research taste and problem formulation ability
“There's part of the scientific process. I think that the models haven't been imbued with yet. And I'm sure people are thinking about how to do that. You know, like what, trying to get to what is the right question as opposed to here's a well-defined thing and …”
Dan Roberts expects more AI-driven math and science breakthroughs within six months
“For the next six months. Like, I think we'll see more of these sorts of math and science breakthroughs.”
AI systems evolving into fully fledged scientists will be a gradual transition
“The, there's no sharp point, or I don't think there will be a sharp point where we'll say that systems didn't, weren't able to be useful for scientific, the scientific process to their fully fledged scientists. There'll be sort of a gradual shift.”
OpenAI publishes most math research results using informal language settings
“Most of our results that we publicize, as far as I can think, are all in the informal setting.”
Roberts: Powerful pre-trained models are necessary for effective RL and reasoning
“If you have a powerful enough pre-trained model, then it can start to do well at RL. It can start to like think at use test time compute to for instance, solve, solve math problems that it wouldn't otherwise be able to do.”
Roberts: AI models improve performance by generating running thought tokens in language
“The natural way it thinks is in language. It's a language model, and so that's sort of this key insight that, that you can cause it to do better just by producing a thought process in, in, in token space, in, in language.”
OpenAI will release reinforcement learning products for consulting, banking, and legal
“I definitely think OpenAI will have amazing products that will be relevant in those domains, and some amount of RL will play a role in there.”
Continuous AI improvements render multi-year autonomous agent runs highly inefficient
“In, in general, we're not just going to like set up a system and let it think autonomously for eight years, if anything, because like the system's eight years. After will be so much more powerful that it probably doesn't make sense to let a system think for a …”
OpenAI studied reinforcement learning reasoning models internally long before releasing o1
“So before we released a one and thinking reasoning models, we were studying this internally”
OpenAI's RL team researches models two generations ahead of current releases
“And to do that, we need to make thinking models and some, somewhere along the way, we interact with that process. Usually at the earlier stage for models, you know, not the next model, but things that are like the next model or the next, next model.”
One line per show, most statements first. The link opens Dan's full record on that show: the calibration, argument clarity, speaking style and every statement made there.
| Show | Role there | Eps | Statements | Record | |
|---|---|---|---|---|---|
 MADLEDGER | Research Lead, OpenAI | 1 | 17 | 100% 1/1 | full record on the MAD Podcast → |