Jan 11, 2024 · 1h 35m · latent-space
The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space Podcast, AI researcher Dr. Nathan Lambert delivers a comprehensive masterclass on Reinforcement Learning from Human Feedback (RLHF), exploring its mathematical mechanics, historical lineage, data economics, and emerging alternatives like Direct Preference Optimization (DPO). The discussion provides critical technical insights into how alignment steers frontier language models, the transition toward synthetic supervision, and the maturation of open-source alignment ecosystems.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 4.6 Guest teaching 5.3 Guest disagreement 1.4 The hosts pushing back 0.8
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:16:00 Nathan critiques academic DPO researchers dodging intuitive explanations
Nathan candidly criticizes how academic authors deflect practical questions by telling people to just stare at mathematical equations on poster boards.
Hardest push from the hosts ▶ 1:10:20 Swix challenges dismissive view on weak-to-strong generalizationWhen Nathan brushes aside OpenAI's weak-to-strong paper as potential safety-washing, Swix firmly intervenes to articulate the exact conceptual lineage connecting Constitutional AI to superalignment.
Biggest teaching moment ▶ 1:06:00 Nathan demystifies how Constitutional AI actually operatesNathan breaks down how Constitutional AI is widely misunderstood, educating the hosts on its two distinct phases: principle-based instruction revision followed by sampling constitutional prompts.
The host holds their own ▶ 38:46 Swix introduces Arrow's Impossibility Theorem into the RLHF frameworkSwix demonstrates advanced interdisciplinary domain knowledge by directly connecting welfare economics' Arrow Impossibility Theorem to the foundational flaws of pairwise preference aggregation.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Ultra-Endurance Sports and Staying Grounded | 3 | 1 | 1 | 0 | Casual introductory rapport where Swix and Nathan discuss ultra-endurance running, trail access in Berkeley, and Nathan's Substack essays. | |
| Motivation for the RLHF Presentation | 4 | 4 | 1 | 1 | Alessio and Nathan discuss how RL transitioned from traditional robotics to LLM alignment, citing the Llama 2 paper's surprise at RL's efficacy. | |
| The Historical and Philosophical Roots of RLHF | 3 | 6 | 1 | 0 | Nathan details the philosophical and decision-theoretic assumptions underpinning RLHF, citing Aristotle and the Von Neumann-Morgenstern utility theorem. | |
| Bridging Social Sciences and AI Engineering | 4 | 5 | 2 | 1 | Alessio inquires about the intersection of sociology and AI systems engineering, while Nathan explains why RLHF formulation resembles a bandit problem more than traditional RL. | |
| Chain-of-Thought Reasoning and Process Reward Models | 4 | 6 | 1 | 0 | Nathan breaks down process reward models and historical milestones from TAMER (2008) to the Christiano et al. (2017) human preference paper. | |
| Signal Richness and Negative Feedback in Alignment | 6 | 4 | 1 | 2 | Swix highlights the distinction between next-token prediction and negative feedback in RLHF; Nathan agrees it is a richer, more compute-efficient supervisory signal. | |
| The Core Assumption: Evaluating vs. Generating Text | 4 | 5 | 1 | 0 | Nathan reviews the core assumption from OpenAI's 2018–2020 papers that evaluating text is significantly easier than generating high-quality text. | |
| Instruction Tuning vs. RLHF in Practical Applications | 5 | 6 | 2 | 1 | Nathan advises startups against building full RLHF pipelines unless critical to their moat, advocating instruction tuning and DPO instead. | |
| The Power and Limits of Instruction Tuning at Scale | 5 | 5 | 1 | 1 | The conversation touches on early open-source instruction tuning milestones like Vicuna, contrasting small model limits against 70B parameter scaling. | |
| Mathematical Formulation of the RLHF Objective | 4 | 7 | 1 | 0 | Nathan explains the mathematical RLHF objective, detailing the role of the KL-divergence constraint to prevent model mode collapse and reward hacking. | |
| Challenges in Preference Aggregation and Annotation Guidelines | 7 | 4 | 1 | 2 | Swix introduces Arrow's Impossibility Theorem regarding preference aggregation; Nathan acknowledges social choice limitations and explains how annotator guidelines try to bypass it. | |
| Preference Data Categories and Granular Metadata | 6 | 5 | 1 | 2 | Swix asks whether safety guardrails should be handled in RLHF or via downstream classifier filters, prompting Nathan to outline industrial deployment trade-offs. | |
| The High Economic Cost of Human Preference Data | 6 | 5 | 1 | 1 | Swix and Nathan break down the massive economic costs of human preference data versus synthetic data generation via GPT-4. | |
| Open Preference Datasets and LMSYS Chatbot Arena | 4 | 6 | 1 | 0 | Nathan describes the Bradley-Terry preference loss function math and the 65-75% inter-annotator agreement ceiling observed across industry datasets. | |
| Implementing Proximal Policy Optimization (PPO) for LLMs | 3 | 7 | 2 | 0 | Nathan deconstructs the multi-model architecture needed for PPO (actor, critic, reference, reward models), emphasizing systems engineering overhead. | |
| Impact of RLHF on Benchmark Capabilities vs. Style | 5 | 5 | 2 | 1 | Nathan points out that RLHF mostly alters model tone and style rather than factual academic capabilities, citing GPT-4 technical report exam charts. | |
| Non-PPO Alternatives: Best-of-N and Rejection Sampling | 4 | 6 | 1 | 0 | Nathan reviews lightweight non-PPO optimization strategies including Best-of-N inference sampling and iterative rejection sampling. | |
| Unpacking Anthropic's Constitutional AI | 4 | 8 | 2 | 1 | Nathan clarifies Anthropic's Constitutional AI, correcting common misconceptions by explaining its dual-phase nature of instruction revision and principle sampling. | |
| Superalignment and Weak-to-Strong Generalization | 7 | 3 | 2 | 4 | When Nathan notes skepticism around OpenAI's weak-to-strong generalization paper, Swix pushes back with a comprehensive explanation connecting it to superalignment. | |
| Direct Preference Optimization (DPO) Mechanics & Debate | 5 | 6 | 3 | 1 | Nathan explains DPO mechanics and implicit reward formulations, playfully critiquing academic researchers who avoid explaining the intuitive mechanics. | |
| AI2's Mission and Scaling TÜLU 2 via DPO | 4 | 5 | 1 | 0 | Nathan details the Allen Institute's mission and the training run behind Tulu 2 70B using JAX TPUs and DPO recipes. | |
| Evaluating LLMs: Leaderboards and Real-World Usage | 5 | 5 | 2 | 1 | Nathan critiques automated leaderboard gaming and explains the significance of LMSYS Chatbot Arena human Elo rankings. | |
| Comparing Chat Evaluation Benchmarks: MT-Bench vs. AlpacaEval | 4 | 6 | 1 | 0 | Nathan contrasts the evaluation methodologies of MT-Bench (multi-turn GPT-4 scoring) and AlpacaEval (pairwise win rate versus DaVinci-003). | |
| Open Frontiers and Outer-Loop Optimization in RLHF | 5 | 6 | 1 | 0 | Nathan outlines outer-loop RLHF optimization (continual retraining on user prompts and fixes) as described by John Schulman, wrapping up with startup landscape analysis. |
Statements from this episode (57)
Lambert: OpenAI's rumored Q* was likely just a moderate benchmark bump
“They probably just got like a moderate bump on one of their benchmarks, and then everyone lost their minds, so it doesn't really matter.”
Lambert: AI field hasn't learned much more, just articulates ignorance better
“I think it's, I try to do it every six or 12 months is my current, is my estimated cadence, just to refine the ways that I say things, and people will see That we don't know that much more, but we have a bit of better way of saying what we don't know.”
Lambert: Major tech companies now realize they need dedicated RLHF teams
“I think any major company that wasn't doing RLHF is now realizing they have to have a team around this.”
Lambert: Open-source and research communities lack large-scale RLHF capabilities
“At the same time, we don't have a lot of that in the like open and research communities at the same scale.”
Lambert: Anthropic is considered the master of RLHF techniques
“Everyone knows Anthropik is kind of the masters of this”
Nathan Lambert: RL will remain a distinct field from language modeling
“I think in the long run it will still settle out, or RL will still be a field that people work on just because of these kind of fundamental things that I talked about, that it's just viewing the whole problem formulation different than predicting text, really,…”
Lambert: Reinforcement learning in language models is contrived and not real RL
“And the view of RL in language models is pretty contrived already. So it's not like we're doing real RL.”
Lambert: RLHF reward models lack inductive biases for true preferences
“In RLHF calling things a preference model is a little annoying because There's no inductive bias of what a preference is.”
Lambert: Claude's constitution dictates output priorities, not model beliefs
“If you look at Claude's constitution, like, that doesn't mean the model believes these things. It's just trying Trained and to prioritize these things.”
Lambert: Frontier AI labs do not recruit economics or social choice academics
“The RLHF techniques that people use were built in, like, labs like OpenAI and DeepMind, where there are some of these people, they have, they, these places do a pretty good job of trying to get these people in the door when you compare them to, like, startups,…”
Lambert: Real-Time Human Feedback in LLM RL Loops Is Far From Feasible
“Setting up the infrastructure to take tens of thousands of prompts and generate them and then show them to a human and collect the human responses and then show that Shove that into your training architecture is very far away from working, so we don't really h…”
Lambert: RLHF operates as a contextual bandit problem, not sequential RL
“What happens is the state is a prompt, and then you do a completion, and then you throw it away, and you grab a new prompt. Where really in like RL, you, as an RL researcher, you would think of this as being like, you take a state, you complete, Get some compl…”
Lambert: AI community will clarify if chain-of-thought maps to RL within a year
“I think in the next year that'll probably get kind of made more concrete by the community on, like, if you can easily draw out, like, if chain of thought reasoning is more like RL.”
Lambert: RLHF provides a richer signal per unit of compute than pre-training
“As reinforcement learning is so much less compute, like it, Is a richer signal in terms of its impact, because if they could do what RLHF is doing at pre-training, they would, but they don't know how to have that effect in like a stable manner. Otherwise, ever…”
Lambert: Instruction tuning is more important than RLHF for most practitioners
“I think for most people, instruction tuning is probably still more important in their day-to-day life. I think instruction tuning works very well. You can write samples by hand that make sense. You can get the model to learn from them. You could do this with v…”
Lambert: DPO benchmark gains rely largely on the UltraFeedback dataset
“Everyone's using this ultra feedback data set and it boosts AlpacaVal, MTBench, TruthfulQA, and like the qualitative model a bit. We don't really know why.”
Lambert: Startups should avoid RLHF unless it offers niche advantage
“I don't really recommend most startups to do it unless it's like going to provide them a clear competitive advantage in their kind of niche. Yeah. Because you're not going to make your model ChatGPT like better than OpenAI or anything like that.”
Lambert: Open-source claims of ChatGPT-level performance are overblown
“I think the claims of ChatGPT level are long overblown in most of the things in open source.”
Lambert: Scaling from 7B to 70B parameters fixes nuance and repetition
“I think the things that people see now is like the small models don't really handle nuance as well, and they could be more repetitive if, even if they have really good instruction tuning, but if you take that kind of seven to seventy billion parameter jump, li…”
Lambert: The vast majority of instruction tuning data remains simple Q&A
“There's much more, like there's surely kind of more tricky things that people do, but I still think the vast majority of it is question and answer. It's like, please explain this topic to me, generate this thing for me. That hasn't changed that much this year.…”
Lambert: RLHF fails completely without a KL divergence constraint
“The KL constraint prevents that. There's not that much documented work on that, but there's a lot of people that know if you take that away, it just doesn't work at all.”
Lambert: Training LLM reward models on 0-to-10 ratings failed
“People tried that with language models, which is if you have a prompt and a completion and you just have someone rate it from zero to 10, could you then train a reward model on all of these completions and zero to 10 ratings and see if you could actually chang…”
Lambert: RLHF avoids controversial preferences, focusing on correctness and style
“The reason this really is done on a deep level is that you're not actually trying to model any, like, contestable preference in this. Like, you're not trying to go into things that are controversial or anything. It's really, like, the notion of preference is t…”
Lambert: Annotator disagreement data in RLHF is not used for anything
“I don't really think this disagreement data is used for anything, but it's good to know, like, what the distribution of prompts is, who's doing it, how many samples you have, controlling the workforce.”
Lambert: OpenAI trains its RLHF models primarily on good-vs-good answer comparisons
“I think open AIs of the world are all in good answer, and have learned to eliminate everything else.”
Lambert: Anthropic, ChatGPT, and Bard Use Post-Generation Moderation Classifiers
“Anthropic and ChatGPT and Bard almost surely have a classifier after, which is like, is this text good? Is this text bad?”
Lambert: A 100x smaller language model filters output better than RLHF
“You could use like a hundred times smaller language model and do much better at filtering than RLHF”
Lambert: Meta spent roughly $6M to $8M on Llama 2 preference data
“So I would say, still say, like, six to eight million is safe to say that they're spending, if not more, they're probably also buying other types of data and or throwing out data that they don't like.”
Lambert: Published AI training compute costs understate total experimentation budgets
“The compute costs listed in the paper always are way lower, because all they have to say is, like, what is one run cost, but they're running tens or hundreds of runs, so it's like, okay, like, it's kind of a meaningless number, yeah.”
Lambert: Open source will learn to train models on arbitrary preference data
“I really think people in open source and academics are going to figure out how to use any preference data on any model just because they're scrappy.”
Lambert: Only 20% to 40% of Meta's Llama RLHF data is useful
“I do think that if we had all the llama data, we wouldn't know what to do with all of it. Like, probably, like, 20 to 40% would be pretty useful for people, but not the whole data set. Like, a lot of it's probably kind of gibberish, because they had a lot of d…”
Lambert: GPT-4 achieves 80% preference labeling agreement versus 70% for humans
“Essentially, people also think that synthetic data is, like, GPT-IV is more accurate than humans at labeling preferences, so if you look at these diagrams, like, humans are about 60 to 70% agreement, or, like, that's what the models get to, and if humans are a…”
Lambert: OpenAI Will Not Aggressively Ban Synthetic Training Scraping
“I don't expect OpenAI to go too crazy on this, because they're just gonna, there's gonna be so much backlash against them.”
Lambert: Practitioners Will Adopt Constitutional AI for Preferences in 2024
“I think in twenty-twenty-four at some point people will start doing things like constitutional AI for preferences.”
Lambert: Zephyr was the first open model to succeed with public RLHF
“I think Zephyr was the first model that showed success with RLHF in the public, but that's a long time from everyone knowing that it was something that people are interested in to having any, like, check mark.”
Lambert: Chatbot Arena is the best available evaluation benchmark for LLMs
“I have, if we make it to evaluation, I'd pretty much say that Chat Arena is the best limited evaluation that people have to learn how to use language models, and like, It's very valuable data”
Lambert: Anthropic and OpenAI reward model loss functions are mathematically identical
“Fun fact is that these loss functions Look different and anthropic in opening eyes papers, but they're just literally just log transform. So if you start like expantiating both sides and taking the log of both sides, you'll like converge on one of the two, the…”
Lambert: RLHF reward models achieve only 65% to 75% validation agreement
“If you look at a test set, you'll have a chosen and rejected, and you can take the reward model you're training, pass in those completions, And you see if the chosen predicted reward, so the scalar number is higher than the rejected predicted reward, and this …”
Lambert: RLHF Performance Depends on Data and Systems Over RL Details
“It really ends up being kind of, like, gibberish that I think is less important now, because it's more about data and infrastructure than RL details, than, like, value functions and everything. A lot of the papers have different terms in the equations. I think…”
Lambert: Reinforcement Learning Does Far Less for Alignment Than People Think
“It's like, now you're getting to the point where you don't even really need this to get a good model, so that's why it's like, okay, the RL is such a small part of the actual, like, doing RLHF. Like, RLHF is a metaphor for, like, all language model adaptation,…”
Lambert: RLHF has not been shown to improve underlying model benchmark capabilities
“RLHF is not that shown to improve capabilities yet. I think one of the fun ones is from the GPT-IV technical report. They essentially listed their kind of bogus evaluations, because it's a hilarious table, because it's like LSAT AP exams, and then like AMC-X a…”
Lambert: Meta Used Rejection Sampling to Bootstrapping Llama 2 RLHF
“Llama started their RLHF process with this to get some signal out of preference data. That preference data went into a reward model, and then the reward model did a good enough ranking that it was, like, essentially superpowered instruction tuning based on rew…”
Lambert: Offline RL could take off due to simpler training pipelines
“There's a few papers that people have published
Not a lot of traction.
I think it could take off.
Some people that I know in the RLHF area really think a lot of people are doing this in industry just because it makes the kind of training process simpler and th…”
Lambert: RL feedback mechanisms will specialize across distinct task domains
“It seems very likely that different feedback will be used for different domains.
Chain of thought reasoning is great.
For math, and that's where these process reward models are being designed.
Probably not great for things like poetry, but as any tool gets bet…”
Lambert: Most open-source RLHF training runs only last a few epochs
“Most RLHF is only a few epochs, at least in the open models”
Lambert: Anthropic Constitutional AI and OpenAI Superalignment share intellectual roots
“The constitutional AI and the super alignment is, like, very conceptually linked. It's like a group of people that has, like, a very similar intellectual upbringing, and they work together for a long time, like, coming to the same conclusions in different ways…”
Lambert: DPO is closer to RLHF than RLHF is to RL
“I think
DPO is closer to RLHF than RLHF is to RL.”
Lambert: More DPO models will emerge than any other method
“I expect to see more DPO models than anything else in the next six months.”
Lambert: AI2 trained 70B TÜLU 2 on the first run without ablations
“Let's just try the Zephyr recipe on seventy billion parameters, and it's literally, like, the first run. It's like, we did no ablations, didn't change any parameters, we just copied them all over. And like, that's the model that people have been working with”
AI2 Plans to Release Fully Open Pre-Trained LLMs With Data and Code
“The Allen Institute is training, pre-training language models, or pre-training, like, open language models, where we'll be able to share, like, data, code, everything, the kind of horn that everyone likes to get annoyed about these days, it's like, well, I'm n…”
Lambert: DPO Has Become the Standard Release Expectation for Open-Source LLMs
“I think DPO releases are kind of becoming expected because Mistral released a DPO model as well. I think the slide after this is just like, there's a ton. It's like Intel releases DPO models, Stability releases DPO models. At some point, you just have to accep…”
Lambert: GPT-4 Turbo Showed a Noticeable Jump on LMSYS Chatbot Arena
“GPT-IV Turbo is also notably ahead of the other GPT-IVs, which it kind of showed up immediately once they added it to the leaderboard, or to the arena, and I was like, all the GPT-IV memes aside, it seems like this is effectively a bump in the model.”
Lambert: GPT-4 Turbo Gap Over Original GPT-4 Exceeds TÜLU 2 to GPT-4 Gap
“So it's like the difference from these, the GPT-IV Turbo to like the GPT-IV that was first released is bigger than the difference from Tulu-II to GPT-IV.”
Lambert: Frontier Labs Lack Visibility into Cross-Model RLHF Sensitivity
“I think big labs are so over-indexed, are indexed on their own base models, so they don't know, like, what's swapping between CloudBase or GPT-IV-Base, how that would change any notion of preference or what you do with RLHF.”
Lambert: OpenAI retrains reward models with curated and user prompt mixtures
“And this is like a sort of outer loop optimization that no one in the open is even remotely qualified to talk about, but OpenAI does monitor and they'll like rerun RLHF and train a new reward model with a mixture of their curated data and user prompts to try t…”