Assertion Supported AI assessment confidence: 98% certainty 4/5 debate potential 1/5

Lambert: Anthropic and OpenAI reward model loss functions are mathematically identical

Nathan Lambert · The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert · Jan 11, 2024 · at 54:41

Dr. Nathan Lambert, AI alignment researcher, explains the mathematical formulations behind training language model reward models in RLHF.

0:00 / 0:14exact quote · 14.2s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“Fun fact is that these loss functions Look different and anthropic in opening eyes papers, but they're just literally just log transform. So if you start like expantiating both sides and taking the log of both sides, you'll like converge on one of the two, the both the two papers end up being the same thing.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

The reward model loss functions published in OpenAI and Anthropic research papers are mathematically identical under a log transform.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

OpenAI's published reward model loss (e.g., Stiennon et al., 2020; Ouyang et al., 2022) is formulated as -log(sigma(r_w - r_l)), where sigma is the logistic sigmoid function sigma(z) = 1 / (1 + e^(-z)). Anthropic's published preference model loss (e.g., Askell et al., 2021; Bai et al., 2022) is written as log(1 + e^(r_bad - r_good)). Because -log(1 / (1 + e^(-Delta r))) = log(1 + e^(-Delta r)), substituting Delta r = r_w - r_l demonstrates that the two loss functions are algebraically identical representations of the negative log-likelihood under the Bradley-Terry preference framework.

OpenAI (Stiennon et al., 2020 / Ouyang et al., 2022) defines RM loss as -E[log sigma(r(x, y_w) - r(x, y_l))].
Anthropic (Askell et al., 2021 / Bai et al., 2022) defines PM loss as log(1 + e^(r_bad - r_good)).
Algebraic expansion: -log(sigma(Delta r)) = -log(1 / (1 + e^(-Delta r))) = log(1 + e^(-Delta r)) = log(1 + e^(r_bad - r_good)), proving exact mathematical equivalence.

Checked against

[1] neurips.cc [2] arxiv.org [3] ainiketan.in [4] wikipedia.org [5] arxiv.org [6] arxiv.org [7] arxiv.org [8] arxiv.org

"loss = -log" "sigmoid" "Askell" "preference model" OR "cross-entropy" "The preference modeling loss for each pair" "Askell" OR "2112.00861" "Training a Helpful and Harmless Assistant with Reinforcement Learning" "loss" "preference model" formula Stiennon et al 2020 Learning to summarize with human feedback reward model loss Ouyang et al 2022 InstructGPT Training language models to follow instructions with human feedback loss function reward model Bai et al 2022 Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback loss function reward model "Training a Helpful and Harmless Assistant" "loss" "PM" "log(1 + e^" OR "\sigma" OR "r(x,y)" "A General Language Assistant as a Laboratory for Alignment" "preference model" "loss" "Bai et al." "preference model" "r(x, y)" "log" "sigmoid" OR "softmax" "loss"

assessed as of 2026-09-03

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

Mentioned

Anthropic OpenAI reward model loss function

⚑ Spot an error? Report it

More from Nathan Lambert

Assertion Supported

Lambert: Tulu 3 matches or beats Meta Llama 3.1 on core evals

“On, like, core evals for our Suite of models from, I think, eight, seven D and four or five B is based on llama at the time. It's like it matches or beats meta on these core valves.”

Nathan Lambert Jul 31, 2025 ▶ 2:20 The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Opinion

Lambert: Deep Research relies on modular RL tasks rather than end-to-end outcomes

“I think the deep research blog post kind of hints that they do a bunch of small scale RL and then poof, the system works. Which I think is much more of what's happening is people train on a bunch of small things and they do some prompting and they see that whe…”

Nathan Lambert Jul 31, 2025 ▶ 7:35 The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Insight

Lambert: RLHF can never be permanently solved

“In the same way that chatbot arena can never be saturated. RLHF can never be solved.”

Nathan Lambert Jul 31, 2025 ▶ 16:42 The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Insight

Lambert: The RL algorithm is not the most important component in reasoning models

“I definitely don't think the algorithm tends to be the most important thing.”

Nathan Lambert Jul 31, 2025 ▶ 19:28 The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Prediction Not checkable as stated

Lambert: Hybrid reasoners may be phased out except for niche uses

“I think in plenty of ways, like hybrid reasoners might just be aged out except for niche applications because quality is so much more important than having a hundred X less inference tokens. It's like you just pay for it and compute and that'll get better.”

Nathan Lambert Jul 31, 2025 ▶ 20:52 The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Insight

Lambert: SFT cannot teach emergent tool use; models must learn via RL environments

“It's very easy to get the model to do tools if you prompt it to, but it's very hard to get the like RL model to learn that the tool is useful. And that's why it's to go through these things where it's like 80 failed tool uses and it still gets it or like it st…”

Nathan Lambert Jul 31, 2025 ▶ 24:35 The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Lambert: Anthropic and OpenAI reward model loss functions are mathematically identical

More from Nathan Lambert

Turn any episode into a week of clips.