Jan 2, 2025 · 16m · latent-space
The State of Reasoning — from Nathan Lambert, Interconnects/AI2 [LS Live @ NeurIPS 2024]
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this NeurIPS 2024 presentation, Nathan Lambert explores the state of artificial intelligence reasoning, examining how chain-of-thought prompting, verifiable post-training reinforcement learning, and reinforcement fine-tuning drive frontier problem-solving capabilities.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 0.0 Guest teaching 0.0 Guest disagreement 1.8 The hosts pushing back 0.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:40 Dismissing reasoning skepticism as ridiculous
Lambert forcefully rejects skepticism claiming language models cannot reason, calling debates over human-like definitions unnecessary and ridiculous.
Hardest push from the hosts ▶ 0:07 No host pushback (Monologue)The episode consists entirely of a solo talk given by Lambert at NeurIPS, resulting in zero host pushback across the recording.
Biggest teaching moment ▶ 5:30 Demystifying o1 architecture against community hypeLambert educates the room by debunking widespread speculation regarding PRMs and MCTS, explaining that o1 relies on large-scale RL with verifiable outcomes.
The host holds their own ▶ 0:07 No host participation (Monologue)Because the audio is a standalone presentation without an active interviewer, there are no host counterarguments or demonstrations of expertise.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Defining Reasoning and Challenging Human-Centric Benchmarks | 0 | 0 | 3 | 0 | Lambert delivers a solo talk challenging standard human-centric definitions of reasoning, criticizing public discourse around LLM limitations as ridiculous. The format is a monologue with zero host participation. | |
| Chain of Thought as Intermediate Compute | 0 | 0 | 2 | 0 | Lambert breaks down how chain-of-thought acts as intermediate variable compute in forward-pass token streams. The monologue continues without any host dialogue. | |
| Decoding OpenAI's o1 and Open Replications | 0 | 0 | 3 | 0 | Lambert dispels speculative community theories about OpenAI o1 utilizing PRMs or MCTS, arguing it is fundamentally massive RL on verifiable outcomes. As a solo conference presentation, host scores remain zero. | |
| The Emergence of Reinforcement Fine-Tuning | 0 | 0 | 1 | 0 | Lambert explains the mechanics of reinforcement fine-tuning (RFT), highlighting how small amounts of RL on top of strong base models avoid performance degradation. No host is present. | |
| Data Formats and Grader Models in RL | 0 | 0 | 1 | 0 | Lambert explains prompt-answer data formatting and the role of grader models and LLM judges in reward shaping. The segment is purely instructional. | |
| Empirical Evaluation Results and Concluding Remarks | 0 | 0 | 1 | 0 | Lambert shares empirical evaluation graphs from AI2 research and invites the audience to ask questions. There is no host involvement. |
Statements from this episode (11)
Lambert: AI is developing new reasoning modes that look less human
“I think a big Trend of the year is that we're seeing new types of language model reasoning that look less human and that Can be good for kind of separating the discourse for expecting a really narrow type of behaviors.”
Lambert: Autoregressive LLMs lack explicit internal structures for intermediate state
“Language models have no ability to do this. They are. Kind of per token computation devices where each token is outputted after doing this forward pass and within that there's no explicit structure to hold these intermediate states.”
Lambert: OpenAI's o1 uses token streams as intermediate state compute
“Why oh, one is exciting is because it's a new type of language models that are going to maximize on this view of reasoning, which is that chain of thought in kind of a forward stream of tokens can actually do a lot to achieve better outcomes when you're doing …”
Lambert: OpenAI o1 uses large-scale RL on verifiable outcomes, not MCTS
“You should take open AI at their face value, which they are doing very large scale RL on the verifiable outcomes is what I've added, especially in context of the RL API that they've released, which I'll talk about more, but most of the reasons to believe in mo…”
Lambert: Early DeepSeek and Qwen reasoning models are substantially narrower than o1
“And I think that these models are really substantially narrower than these full O-one models from OpenAI. So OpenAI is, if you use O-one, you can do it for a lot more tasks. If you use, like I was using the DeepSeq model, and it's supposed to be for math or co…”
Lambert: Open community will eventually match OpenAI's large-scale RL infrastructure
“And this is something that these early relative models are not going to be doing because we don't like, no one has this infrastructure like open AI does. It'll take a while to do that, but people will make it.”
Lambert: Reinforcement fine-tuning will succeed where answer correctness matters over style
“It is just a new paradigm for fine tuning, and I have seen some of this work, and I'm pretty Optimistic that it'll work for kind of kind of really specific capabilities where answers matter rather than features in your style of text mattering.”
Lambert: Reinforcement fine-tuning requires only dozens of labeled samples
“This reinforcement fine tuning does many passes over the data, which is why they can say you only need dozens of labeled samples to actually learn from it, which is very different than. Previous training regimes”
Lambert: Llama 3.1 math evals rely on SymPy and LLM judges
“Lama, 3.1 details their vows for math. They use both SIM by a Python process or Python package for extraction and it's a judge to extract their answers for math.”
Lambert: Open judge models will become core open RL infrastructure
“We already have a bunch of open models that are doing like judge of models and Prometheus and other things that are designed specifically for LM as a judge. And I see that continuing to just become part of this kind of open RL infrastructure.”