Assertion Supported AI assessment confidence: 95% certainty 4/5 debate potential 2/5

Kaiser: Reinforcement learning causes AI models to self-correct mistakes

Łukasz Kaiser · What’s Next for AI? OpenAI’s Łukasz Kaiser (Transformer Co-Author) · Nov 26, 2025 · at 20:59

Łukasz Kaiser, Lead Research Scientist at OpenAI, describes how reinforcement learning training induces self-verification and error correction behavior in reasoning models.

0:00 / 0:24exact quote · 24.1s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“Even for math and coding, you start seeing that the models start correcting their own mistakes, right? Earlier, if the model made a mistake, it generally just tell you what it did and insist that the mistake was right or something like that. With the thinking, it's like, oh, I often make mistakes, but I need to verify and correct myself to give the correct answer. So, so this just emerges from this reinforcement learning.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

Reinforcement learning for chain-of-thought reasoning causes AI models to self-correct mistakes rather than insisting an incorrect answer is right.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

Multiple technical reports and peer-reviewed studies confirm that reinforcement learning (RL) applied to chain-of-thought reasoning causes LLMs to develop emergent self-correction and verification capabilities. OpenAI's o1 series, Google DeepMind's SCoRe framework, and DeepSeek-AI's DeepSeek-R1 technical report demonstrate that RL training incentivizes models to backtrack, recognize mistakes during long CoT generation, and correct errors rather than persisting in incorrect paths.

OpenAI o1 System Card (2024): RL training enables the model to refine its chain-of-thought, recognize errors, and backtrack.
Google DeepMind SCoRe (arXiv:2409.12917, Sept 2024): Demonstrated multi-turn RL successfully enables models to self-correct errors on reasoning benchmarks like MATH and HumanEval.
DeepSeek-R1 Technical Report (arXiv:2501.12948 / Nature 2025): Showed pure RL incentivizes emergent self-reflection, verification, and self-correction ('aha moments') in CoT generation.

Checked against

[1] openai.com [2] arxiv.org [3] arxiv.org [4] arxiv.org [5] huggingface.co [6] openreview.net [7] researchgate.net [8] aclanthology.org

DeepSeek R1 technical report reinforcement learning chain of thought self correction OpenAI o1 technical report reinforcement learning self correction chain of thought reinforcement learning chain of thought emergent self correction LLM research paper evaluating self-correction in reasoning models DeepSeek R1 o1

assessed as of 2026-07-30

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

Mentioned

OpenAI Łukasz Kaiser

⚑ Spot an error? Report it

More from Łukasz Kaiser

Assertion Not checkable as stated

Kaiser: Pre-training between GPT-4 and GPT-5 focused on reducing costs

“The pre-training part in that timeframe was mostly about making things cheaper. Not making things better.”

Łukasz Kaiser Nov 26, 2025 ▶ 40:36 What’s Next for AI? OpenAI’s Łukasz Kaiser (Transformer Co-Author)

Insight

Kaiser: Reasoning models are the second major milestone after Transformers

“One point was, of course, the Transformers when it started, but the other point was reasoning models.”

Łukasz Kaiser Nov 26, 2025 ▶ 3:53 What’s Next for AI? OpenAI’s Łukasz Kaiser (Transformer Co-Author)

Assertion Not checkable as stated

Kaiser: Pre-training scaling laws still hold across OpenAI and Google

“What scaling clause says is that your loss will log linearly decrease with your compute. We totally see that and clearly Google sees that and all other labs.”

Łukasz Kaiser Nov 26, 2025 ▶ 4:35 What’s Next for AI? OpenAI’s Łukasz Kaiser (Transformer Co-Author)

Insight

Kaiser: Pre-training science is plateauing, but compute scaling still improves loss

“Pre-training, as I said, I think it has reached this upper level of the S-curve in terms of science, but it can scale smoothly. Meaning if you put More compute. You will get better losses if you do things right, which is extremely hard, and that's valuable.”

Łukasz Kaiser Nov 26, 2025 ▶ 33:07 What’s Next for AI? OpenAI’s Łukasz Kaiser (Transformer Co-Author)

Assertion Not checkable as stated

Kaiser: Model hallucinations are dramatically lower than two years ago

“There was these things called hallucinations. It's still with us to some extent, but dramatically less than two years ago.”

Łukasz Kaiser Nov 26, 2025 ▶ 41:53 What’s Next for AI? OpenAI’s Łukasz Kaiser (Transformer Co-Author)

Insight

Kaiser: Test-time compute increases AI capabilities faster than pre-training

“Using more tokens to think increases your capability, and it increases it, given the computation, way faster than pre-training, right?”

Łukasz Kaiser Nov 26, 2025 ▶ 46:49 What’s Next for AI? OpenAI’s Łukasz Kaiser (Transformer Co-Author)

Kaiser: Reinforcement learning causes AI models to self-correct mistakes

More from Łukasz Kaiser

Turn any episode into a week of clips.