Lambert: OpenAI o1 uses large-scale RL on verifiable outcomes, not MCTS
Nathan Lambert · The State of Reasoning — from Nathan Lambert, Interconnects/AI2 [LS Live @ NeurIPS 2024] · Jan 2, 2025 · at 5:23
This episode carries Nathan Lambert's own address, with nobody on the show putting questions to them. It still counts as said, and it is kept out of every score on their page.
AI researcher Nathan Lambert discusses the architectural debates surrounding OpenAI's o1 model at NeurIPS 2024.
0:00 / 0:36exact quote · 36.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“You should take open AI at their face value, which they are doing very large scale RL on the verifiable outcomes is what I've added, especially in context of the RL API that they've released, which I'll talk about more, but most of the reasons to believe in more complicated things like process rewards models, self-play, Monte Carlo tree search are mostly based on. Previous literature and things that we would have expected advanced reasoning to look like for language models and not based on evidence that they have given us or the behavior, whether you're looking at evaluations or how actually like inference is done when serving the model.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Nathan Lambert
Lambert: Tulu 3 matches or beats Meta Llama 3.1 on core evals
“On, like, core evals for our Suite of models from, I think, eight, seven D and four or five B is based on llama at the time. It's like it matches or beats meta on these core valves.”
Lambert: Deep Research relies on modular RL tasks rather than end-to-end outcomes
“I think the deep research blog post kind of hints that they do a bunch of small scale RL and then poof, the system works. Which I think is much more of what's happening is people train on a bunch of small things and they do some prompting and they see that whe…”
Lambert: RLHF can never be permanently solved
“In the same way that chatbot arena can never be saturated. RLHF can never be solved.”
Lambert: The RL algorithm is not the most important component in reasoning models
“I definitely don't think the algorithm tends to be the most important thing.”
Lambert: Hybrid reasoners may be phased out except for niche uses
“I think in plenty of ways, like hybrid reasoners might just be aged out except for niche applications because quality is so much more important than having a hundred X less inference tokens. It's like you just pay for it and compute and that'll get better.”
Lambert: SFT cannot teach emergent tool use; models must learn via RL environments
“It's very easy to get the model to do tools if you prompt it to, but it's very hard to get the like RL model to learn that the tool is useful. And that's why it's to go through these things where it's like 80 failed tool uses and it still gets it or like it st…”