Jul 31, 2025 · 1h 18m · latent-space
The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode, AI2 post-training researcher Nathan Lambert joins hosts Alessio Fanelli and Swix to break down the rise of Reinforcement Learning from Verifiable Rewards (RLVR), the mechanics of test-time compute scaling, and the taxonomy of modern reasoning architectures. Lambert explores agent tool use, reward hacking pathologies, and AI2's strategic roadmap for developing transparent, frontier-grade open-source models.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 26.1% of the talking time here. How this is scored →
The hosts as informed peer 5.5 Guest teaching 4.1 Guest disagreement 1.7 The hosts pushing back 1.9
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 28:17 Inference scaling plots called a psyop
Nathan aggressively rejects the standard industry presentation of test-time compute, labeling the popular inference-time scaling curve a deceptive marketing construct.
Hardest push from the hosts ▶ 41:40 Swyx rejects native plan tokensSwyx directly challenges Nathan's reasoning taxonomy by arguing that modern software engineering workflows favor external tool orchestration over internal model plan tokens.
Biggest teaching moment ▶ 16:47 Why RLHF outlasts RLVR as a research disciplineNathan educates the hosts on why preference tuning and RLHF will remain foundational research challenges indefinitely while verifiable reward RLVR may quickly saturate.
The host holds their own ▶ 52:10 Alessio diagnoses RL code pathologiesAlessio demonstrates deep practitioner expertise by detailing how RL-trained models introduce silent failure patterns through defensive if-statements in real-world codebases.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Welcome and Catching Up with Nathan Lambert | 4 | 3 | 1 | 1 | Friendly intro where Swyx and Alessio establish Nathan's background before Nathan recounts the development of Tulu 3 and the coining of RLVR. The hosts prompt him smoothly with conversational context. | |
| Expanding RLVR into Multi-Hop Tool Environments | 5 | 4 | 1 | 1 | Nathan breaks down the transition from single-generation RLVR to multi-hop environments like o3 and Deep Research. Alessio brings in insights from a previous conversation with Noam Brown regarding non-verifiable rewards. | |
| Agent Progress, Non-Verifiable Tasks, and Data Moats | 5 | 5 | 2 | 2 | Nathan explains why agent progress will look different from raw modeling progress and details the limitations of academic preference datasets compared to frontier lab data moats. Alessio questions whether frontier labs retain a structural edge via live inference inspection. | |
| Evaluating the Longevity of Chatbot Arena Benchmarks | 6 | 3 | 2 | 3 | Swyx presses on whether Chatbot Arena is 'cooked' and discusses vulnerabilities to gaming and the lack of multi-turn evals, citing recent critiques like Sarah Hooker's. Nathan defends the ongoing utility of Elo leaderboards for compression benchmarking. | |
| Writing the RLHF Book Amid the Reasoning Revolution | 5 | 5 | 2 | 2 | Nathan articulates why he is keeping his upcoming book focused on RLHF rather than RLVR, arguing RLHF deals with enduring alignment and human preference problems while RLVR may be solved quickly. Swyx probes his dismissal of GRPO algorithmic hype. | |
| Hybrid Reasoning Models versus Unified Interfaces | 6 | 5 | 2 | 2 | Alessio and Swyx challenge the necessity of hybrid reasoning models versus unified interfaces that self-allocate compute. Nathan outlines why search-augmented models degrade on raw SimpleQA when isolated from external search retrieval. | |
| Deconstructing Q*, o1, and Inference Scaling Plots | 6 | 4 | 3 | 2 | Nathan calls the standard inference-time scaling plot a 'psyop' because plotting test compute on a log axis misleads people into treating search as a simple linear knob. Alessio and Swyx discuss the ARC-AGI benchmark harness controversies. | |
| Strategic Research Opportunities for Academic AI | 5 | 4 | 1 | 1 | Nathan lays out realistic strategies for academic AI institutions like AI2, emphasizing domain-specific evals and artifact production over trying to match frontier labs on raw token burn. | |
| A Taxonomy of Reasoning: Skills, Strategy, Abstraction, Calibration | 6 | 4 | 2 | 3 | Nathan introduces his taxonomy of reasoning: skills, strategy, abstraction, and calibration. Swyx pushes back, questioning whether dedicated plan tokens are necessary when engineers prefer external tool orchestration. | |
| Task Decomposition, Planning Tokens, and Compute Allocation | 5 | 4 | 2 | 2 | Alessio questions whether plan templates could be reused as static blueprints rather than regenerated dynamically on every query. Nathan details how prompt generation remains more cost-effective despite potential abstraction failures. | |
| Parallel Test-Time Compute, Verifiers, and Diffusion LLMs | 6 | 4 | 2 | 3 | Swyx pushes back on Nathan's skepticism of parallel test-time compute, arguing it allows teams to pull forward future model capabilities to generate synthetic data for distillation. | |
| Code Generation Pathologies from Reinforcement Learning | 7 | 3 | 1 | 2 | Alessio brings concrete technical observations from using coding tools, noting that RL induces models to write defensive if-statements to suppress missing environment variables rather than raising clear errors. | |
| The Three Eras of Reinforcement Learning Over-Optimization | 6 | 5 | 1 | 1 | Nathan walks through the three historic eras of RL reward over-optimization across robotics control, RLHF, and RLVR. Swyx asks about the nuances of reward design and partial credit in multi-domain setups. | |
| Infrastructural Constraints and Long Inference Trajectories | 6 | 4 | 2 | 2 | Swyx and Nathan discuss training run limits and wall-clock feedback bottlenecks, referencing Swyx's earlier debate with Noam Brown. Nathan discusses Joanne Jang's Model Spec work at OpenAI. | |
| Micro-Distillation, Hardware Pricing, and AI Wearables | 6 | 4 | 2 | 3 | Alessio and Nathan discuss routing across micro-distilled models for specialized media tasks. Swyx pushes back, claiming multimodal models like GPT-4o will eventually absorb all point solutions, while noting the unexpected appreciation of RTX 4090 GPUs. | |
| Meta's Strategic Pivot and the Economics of AI Talent | 5 | 4 | 2 | 2 | Swyx and Nathan examine Meta's aggressive compensation packages for AI talent, analyzing the economic trade-off between massive cluster compute expenditures and elite researcher retention. | |
| AI2's Roadmap for an Open 'American DeepSeek' | 5 | 4 | 1 | 1 | Nathan outlines AI2's long-term technical roadmap for an open American equivalent to DeepSeek, moving OLMo from dense architectures to sparse MoE reasoning models. |
Statements from this episode (40)
Lambert: Tulu 3 matches or beats Meta Llama 3.1 on core evals
“On, like, core evals for our Suite of models from, I think, eight, seven D and four or five B is based on llama at the time. It's like it matches or beats meta on these core valves.”
Lambert: Academia relied on UltraFeedback for open preference tuning for a year
“The academic community had been using this one data set since like all the way back in the hugging face models of like Zephyr beta is when this ultra feedback data set got popular. And still a year later is like this state of the art data set for open preferen…”
Lambert: RLVR is broader than ground truth because code is verifiable
“The verifiable rewards is actually a more general notion because only like math questions have a ground truth where code is verifiable, precise instruction following is verifiable.”
Lambert: Deep Research relies on modular RL tasks rather than end-to-end outcomes
“I think the deep research blog post kind of hints that they do a bunch of small scale RL and then poof, the system works. Which I think is much more of what's happening is people train on a bunch of small things and they do some prompting and they see that whe…”
Lambert: Context compression is crucial for long-horizon AI agents
“Compressing context, like that's not I don't think that's really a verifiable thing, but that being messed up, like that's a super crucial skill for long context actions and long longer tasks is just compressing well, and that's going to take some training nov…”
Lambert: Frontier AI labs still rely on human preference data
“Every time I check in with people at frontier labs, they're like, yeah, we still use human preference data.”
Lambert: LMSYS is probably setting up a deep research arena
“I mean, they're probably setting up a deep research arena, because that's the data that, I mean, if I was open AI working on deep research, that's the data that I want, and there are competitors, and LMSYS is the entity that has the market placement to set it …”
Lambert: RLHF can never be permanently solved
“In the same way that chatbot arena can never be saturated. RLHF can never be solved.”
Lambert: The RL algorithm is not the most important component in reasoning models
“I definitely don't think the algorithm tends to be the most important thing.”
Lambert: North star of reasoning models is dynamic token budget calibration
“I think that has to be the north star for most people working on reasoning, which is the model will just Spend the right amount of tokens on it.”
Lambert: Hybrid reasoners may be phased out except for niche uses
“I think in plenty of ways, like hybrid reasoners might just be aged out except for niche applications because quality is so much more important than having a hundred X less inference tokens. It's like you just pay for it and compute and that'll get better.”
Lambert: SimpleQA benchmark scores drop across reasoning models tested without tools
“You look at all the evals from reasoning models, and one of the trends is that, like simple QA numbers all drop. It's like DeepSeq R-one to the new R-one, it goes down. It's like all the new, like, QN-II to QN-III, simple QA goes down, at least when you're eva…”
Lambert: All major frontier AI labs will build their own search indexes
“I think they'll all do end up doing their own index and it should, it's one of those things that's like Google should have an advantage again, but who knows if they do.”
Lambert: SFT cannot teach emergent tool use; models must learn via RL environments
“It's very easy to get the model to do tools if you prompt it to, but it's very hard to get the like RL model to learn that the tool is useful. And that's why it's to go through these things where it's like 80 failed tool uses and it still gets it or like it st…”
Swix: OpenAI Deep Research was built by three people as an o3 wrapper
“As far as I know, it's three people did it. It was Isa and like the two other collaborators that she had. I don't know if they did that much on top of all three, like every indication I've had from over the eye is that deep research is more or less a thin wrap…”
Fanelli: Simon Willison reported Anthropic added Brave Search as a sub-processor
“Our friend Simon Willison wrote a post that Anthropic added Brave Search as one of the sub processor in their product.”
Lambert: Inference scaling plots misleadingly suggest search is an easy control knob
“The core of that article is just, they're taking points from within training, or there's a natural variance, and then you line them up. And if you line them up, then you get this nice inference time-scaling behavior, which is, and now people, a lot of people h…”
Lambert: Scaling RL long enough requires a curriculum of increasing difficulty
“If you scale RL long enough,
You're going to need a curriculum of things getting harder. And like, that's pretty obvious.”
Lambert: AI benchmarks like ARC-AGI should prioritize testing without harnesses
“Harnesses are cool, but they're gonna, they're,
They're a handicap that's changing the learning dynamics substantially. So it's good. It's good demos, but I feel like the core thrust has to be no harnesses.”
Lambert: AI academics must build datasets and evals rather than papers
“If you're trying to have impact in AI right now, it's as an academic, you have to like level up out of papers to artifacts, which is models, datasets, evals. Datasets and evals are easier for people to have impact on.”
Lambert: Academics cannot match industry compute on Humanity's Last Exam
“I just think it's kind of unlikely that we're going to win as a academic and a state of the art number because they're going to start spending millions of tokens per query. And it's just a lot of, it's a lot of compute burn. Like the getting, beating that on t…”
Lambert: Reasoning models solved basic skills; planning is the next frontier
“So I came up with four and the foundational one was skills, which is What I would say that we have already done with O-one and R-one, which is you do a lot of RL, you show the inference time scaling works and you get really high benchmark numbers. And then the…”
Lambert: DeepSeek-R1 starts solving math questions immediately without explicit planning
“If you look at DeepSeq R-One and you ask it a hard math question, it's not like, here's my plan of attack. It just starts.”
Lambert: Current Language Models Cannot Prioritize Experiments for Multi-Week Research Plans
“So it's like, how do you come up with a research plan in 10 weeks? Like there's a lot of, how do you prioritize which experiments to do? It's like, there's a lot of inductive biases that go into that, that I don't like a language model would not do well at tha…”
Lambert: Parallel compute provides robustness rather than low-probability search
“Well, I don't think we're using parallel compute in a way to search over like low probability tokens. We're using it to get robustness. If you use like O-one pro is, it was so nice because it Just had a very predictable depth to it, even on niche topics where …”
Lambert: Labs will surely use parallel-compute models to generate synthetic data
“Well, I bet people, I mean, they surely will use these for synthetic data. It's just like the marginal gain on synthetic data is always very high.”
Lambert: Labs trade code usability for massive RL performance gains
“That's just like the labs are trading off massive gains in performance or small detriments in usability. And it's like, do you ship that model? Yeah. Like you just ship it and deal with it later, but I'm sure they could, I'm sure that's a fixable thing.”
Lambert: Code maintainability is a human preference problem in RL
“The software stuff is not easy because it's almost like maintainability almost feels like a human preference type issue again, where somebody could look at it and be like, yeah, that's not as good, but adding the heuristic and trading seems very messy.”
Lambert: RLVR is harder to over-optimize on math than code
“For math, it's a bit harder to over optimize, I think. Unless you have tools and the model learns to search and cheat instead of learning math, which I'm sure somebody could see that out in the world, which is like, oh, I'll just find the, you're training. It'…”
Lambert: RLVR on math does not degrade knowledge benchmark performance
“I think part of the intuition of RLVR is that the model is good at knowing which prompt area it is, which is why the models don't get worse on knowledge benchmarks if you're trading on like just math or precise instruction following. So the model just kind of …”
Lambert: Long inference generations break RL infrastructure and require more GPUs
“The inference, high inference length generations definitely just, like, kind of breaks all infrastructure, because there's just so many tokens, there's more opportunity for out of memory or other things to go wrong. So it's like, just on a default, all of your…”
Lambert: OpenAI's Model Spec is more useful than Anthropic's Constitution
“The model spec is much more useful than a constitution because the constitution is like an intermediate training artifact that you give to the training algorithm in order to get the model that you want. It is not necessarily like what model did we, like we don…”
Lambert: Custom personality fine-tuning is open source AI's winning turf
“If open models are to win, part of it could be just, like, everybody can have exactly the model they want. We're serving GPT-IV. It's kind of its thing. You can prompt it, but if fine-tuning is more effective than prompting, everybody can have the model. That …”
Lambert: The local model community is much smaller than assumed
“Like the local modeling community, I think is much smaller than people give it credit for, because most of the use for open models is still in APIs.”
Swix: Nvidia RTX 4090 prices doubled in the past year
“40 and 90 prices have doubled in the last year.”
Lambert: OpenAI's open model will be best-in-class in its size category
“I expected. It'll be best in class for some size Category in some subset of tasks. That's like, OpenAI only does things like that.”
Lambert: Jony Ive and OpenAI hardware will run in the cloud
“I think that thing will run on the cloud. I don't think that'll run local anyways.”
Lambert: Meta withholding its leading benchmark model is bad execution
“But to be a model that claims to be open and then not release the model that is your leading claim is just, like, that is, like, bad execution.”
Lambert: Top AI talent is dramatically cheaper than GPU clusters
“Talent is cheaper than GPUs by a dramatic margin, and At the end of the day, it's like, okay, if we're spending this much, they go to the room and they stare in the mirror and you're like, wait, it might not actually be that ridiculous to spend this money on t…”