Oct 16, 2025 · 1h 16m · mad
How GPT-5 Thinks — OpenAI VP of Research Jerry Tworek
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The MAD Podcast, OpenAI's VP of Research Jerry Tworek speaks with host Matt Turck about the technological leap from standard language models to advanced reasoning systems like o1, o3, and GPT-5. He shares insights into reinforcement learning, OpenAI's internal research culture, and the technical roadmap driving the pursuit of artificial general intelligence.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 18.8% of the talking time here. How this is scored →
Matt as informed peer 2.9 Guest teaching 3.2 Guest disagreement 0.5 Matt pushing back 0.9
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:13:00 Rejection of Pure RL Thesis
Jerry directly rejects Richard Sutton's pure RL premise, asserting that RL cannot succeed without pre-training.
Hardest push from Matt ▶ 1:12:19 Sutton Pure RL ChallengeMatt challenges OpenAI's hybrid LLM approach by bringing up Richard Sutton's argument that LLMs are merely imitating reality.
Biggest teaching moment ▶ 27:26 Internal IP Transparency CorrectionJerry corrects Matt's intuitive assumption about internal IP compartmentalization, explaining that all researchers at OpenAI have full access to information.
Matt holds his own ▶ 1:12:19 Sutton Philosophical CounterpointMatt demonstrates deep familiarity with AI theory by referencing Richard Sutton's argument on Dwarkesh Patel's podcast regarding pure RL vs LLMs.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Defining Reasoning in Artificial Intelligence | 2 | 3 | 0 | 1 | Matt asks basic foundational questions about what reasoning and chain of thought actually mean in language models. Jerry explains that reasoning is an extended search process to reach unknown answers rather than simple retrieval. | |
| Balancing Thinking Time and User Experience | 3 | 3 | 1 | 1 | Matt synthesizes Jerry's explanation of thinking trade-offs into user experience design. Jerry agrees and candidly notes that O1 was mostly a technology demonstration rather than a fully useful product. | |
| Formative Years and Early Passion for Science in Poland | 1 | 1 | 0 | 0 | Matt asks standard biographical questions about Jerry's early life in Poland. Jerry shares his early dream of becoming a scientist and studying math in Warsaw. | |
| Leaving Academia for Quantitative Trading in Europe | 1 | 1 | 0 | 0 | Jerry describes leaving academia due to disillusionment with its structure, entering quantitative trading at JP Morgan, and co-founding hedge funds in London and Amsterdam. | |
| Discovering Deep RL and Applying to OpenAI | 2 | 2 | 1 | 1 | Jerry reframes the conventional view of AI milestones, stating that 2012 ImageNet wasn't as groundbreaking to him as DeepMind's 2013 DQN results. Matt tracks his move to OpenAI in 2019. | |
| Early Work at OpenAI: Dota 2 and Robotic Rubik's Cube | 3 | 2 | 0 | 1 | Matt asks about OpenAI's transition from early reinforcement learning projects like Dota 2 to unsupervised pre-training. Jerry details his early work on dexterous robotic manipulation solving a Rubik's cube. | |
| A Day in the Life of OpenAI's VP of Research | 2 | 3 | 1 | 1 | Jerry gently rejects Matt's top-down vs bottom-up question framing, explaining that research management at OpenAI requires balancing a tiny number of massive core bets with researcher autonomy. | |
| Internal Transparency and Collaborative Culture | 2 | 5 | 2 | 1 | Matt speculates that OpenAI must strictly compartmentalize IP internally. Jerry immediately corrects him, revealing that all ~600 researchers have access to everything because internal transparency is vital for research quality. | |
| Maintaining Shipping Velocity in a Generational Era | 2 | 2 | 0 | 1 | Matt asks how OpenAI maintains rapid release momentum without sacrificing long-term research. Jerry jokes about paying $200 per month for ChatGPT like any outside user. | |
| Combining Pre-Training with Reinforcement Learning | 3 | 3 | 0 | 1 | Matt accurately frames modern AI as combining pre-training and RL. Jerry recalls Ilya Sutskever stating that exact two-step thesis back in early 2019. | |
| Reinforcement Learning 101: Dog Training Analogy | 1 | 4 | 0 | 0 | Matt asks for a basic high-level explanation of RL. Jerry uses a dog training analogy involving a bag of treats and punishments to explain policy shaping. | |
| Defining Policies, Agents, and Interactive Environments | 4 | 3 | 0 | 1 | Matt lists core technical terms like policy, agent, action, and environment. Jerry clarifies the concept of interactive environments using a guitar-strumming feedback analogy. | |
| Evolution of Deep RL from Games to Pre-Trained Models | 2 | 4 | 0 | 0 | Jerry gives a historical overview of RL, revealing that initial internal reactions to GPT-4 pre-training were underwhelming before post-training techniques were applied. | |
| The Breakthrough of RLHF and Human Preference Data | 3 | 3 | 0 | 1 | Matt asks about RLHF mechanisms and data labeling companies like Scale AI. Jerry explains how the human data labeling industry must constantly shift upstream as models surpass human raters. | |
| Unsupervised Pre-Training and Representation Learning | 4 | 3 | 1 | 1 | Matt raises technical distinctions between self-supervised and unsupervised learning. Jerry pushes back on treating those definitions as stark boundaries, explaining representation learning. | |
| GRPO, DeepSeek, and Industry Reasoning Acceleration | 4 | 4 | 1 | 1 | Matt cites Jerry's recent tweet regarding DeepSeek's GRPO release. Jerry analyzes how DeepSeek's open-source publication accelerated reasoning research across Western AI labs. | |
| Scaling Reinforcement Learning vs. Pre-Training | 3 | 4 | 0 | 1 | Matt asks what scaling RL requires in practice. Jerry contrasts the simple math of pre-training (like a steel mill) with the delicate complexity of scaling RL (like semiconductor fabrication). | |
| Agentic AI and Extended Reasoning Durations | 3 | 3 | 0 | 1 | Matt asks how agentic autonomy intersects with reasoning RL. Jerry outlines internal experiments with models reasoning continuously for 30 minutes to two hours on hard problems. | |
| Online RL and Live User Interaction Risks | 3 | 4 | 1 | 1 | Matt asks if production agents learn in real-time from user interactions. Jerry clarifies the severe safety risks of unconstrained online RL, noting OpenAI avoids real-time user-loop training. | |
| Benchmarking Reasoning in Competitive Programming | 4 | 4 | 0 | 1 | Matt cites OpenAI's impressive results at the ICPC World Finals. Jerry explains that competitive coding benchmark success was an accidental byproduct of using programming puzzles as an internal RL testbed. | |
| Evaluating RL Beyond Coding and Managing Reward Hacking | 4 | 5 | 1 | 1 | Matt asks how RL generalizes to subjective, non-binary fields. Jerry provides a reframe, comparing AI reward hacking directly to human workplace gaming and policymaking incentives. | |
| The Journey Toward AGI and Self-Improving Systems | 4 | 4 | 1 | 1 | Matt quotes Jerry's tweet regarding AGI timelines. Jerry breaks down the psychological shift in AGI definitions and highlights self-improving systems as the key threshold. | |
| Debating Pure RL vs. LLMs and Interview Conclusion | 6 | 4 | 2 | 3 | Matt presents Richard Sutton's argument that LLMs are flawed imitation devices and only pure RL leads to AGI. Jerry directly rejects the premise, arguing pure RL is useless without pre-training. |
Statements from this episode (36)
Tworek: AI reasoning spent on unknown answers yields better results with time
“I think that, that, that difference is here, like answering a question usually means you already know the answer and you just elicit the answer, you know, and the process of reasoning is Getting to the answer that you don't know, and usually the longer you spe…”
Tworek: Calling LLMs strictly next-token predictors is inaccurate in RL era
“Language models do on their own, like fundamental level is they are often called as next token prediction machines. And that's not completely accurate in the age of reinforcement learning, but they still operate on mostly on tokens that are mostly text.”
Tworek: Chain of thought is an LLM's reasoning verbalized in human words
“What chain of thought is, is their thinking process verbalized using human words and human concepts.”
Tworek: OpenAI's high and low reasoning modes use the exact same model
“Where you can have like a high rezoning model and the low rezoning models. And this is like in the end, the same model. You just, we just tweak the parameter, which says we want you to think longer or shorter.”
Tworek: OpenAI's o1 was mostly a tech demo for solving puzzles
“O-one like, to be perfectly honest, it was really mostly good at solving puzzles and like maybe a few kind of thinking problems here and there, but it wasn't like, it wasn't a very useful model. It was almost more like a technology demonstration.”
Tworek: GPT-5 can effectively be considered an iteration like 'o3.1'
“Like GPT-Five in some way I can be considered as like, oh, 3.1. It's a little bit of like, you know, iteration of like the same thing and the same concept”
Tworek: Coding agents are the first successful agentic AI products
“Like, coding agents are at the moment the first, like, pretty successful agentic products built on top of AI.”
Tworek: The landmark 2012 ImageNet AI results were not that significant
“From my perspective, and again, this is just how my brain works, that the 20 12 ImageNet results, like, weren't that significant.”
Tworek: Dexterous manipulation remains an elusive challenge for AI policies
“The project I was working on was focused on dexterous manipulation, which was back then and still continues to be an elusive challenge for trained policies.”
Tworek: OpenAI focuses research on only three or four large-scale projects
“We all work on a very few projects total. There are not that many projects. OpenAI is not trying to do everything. We are not trying to like have portfolio. We are trying to have like multiple different bets. Always the idea is we do a few core things really, …”
Tworek: Top-down management does not work in AI research organizations
“Top down structuring of research doesn't work in research organizations. I really don't believe in it because like you are not kind of hiring some of the smartest people in the world and open air has incredibly, incredibly smart people. To kind of tell them wh…”
Tworek: OpenAI's research division consists of slightly under 600 people
“The truth is in research at OpenAI, which is like around, like slightly less than 600 people at the moment, everyone knows everything really, really it does.”
Tworek: Uninformed researchers pose a greater risk than IP leaks
“It is like, yeah, it is some like risk of losing IP, but I think the risk of not doing the right thing and of people not being informed about research and not being able to do the best research is much higher in my personal opinion and how, how I approach thos…”
Tworek: OpenAI internal teams heavily use Codex for coding
“We definitely use codecs a lot for coding, and this is only getting, getting better.”
OpenAI VP Jerry Tworek pays $200 per month for ChatGPT Pro
“I think I am pretty heavy user of ChatGPT right now, happily paying like 200 dollars a month for it”
Tworek: Ilya Sutskever set OpenAI's current RL research roadmap in 2019
“And what he said at the beginning of 2019 was to train large generative model on all data we can and then do reinforcement learning on it. That was the OpenAI research Plan at the beginning of 2019. And this is exactly what we are doing today.”
Tworek: Effective reinforcement learning requires a 50/50 balance of rewards and punishments
“In a good way, the good way to do RL is if you balance those things. So if you kind of give cookies half of the time and punish the other half of the time, but this is almost like a mathematical kind of kind of aspect of it.”
Tworek: Reinforcement learning is the only way agents learn environmental reaction
“And that's kind of, like, the only way how to, like, really teach agents to, like, learn to react to changes in the environment is through reinforcement learning.”
Tworek: Lack of pre-training was the primary bottleneck for 2019 RL
“Like whenever, even when I started like 20, 2019, the reinforcement learning was kind of fashionable at that moment, although not like very successful, but we were, the reinforcement learning was able to solve a lot of games, but the bottleneck was there that …”
Tworek: OpenAI team was initially underwhelmed by pre-trained GPT-4
“When we trained GPT-IV, we were pretty underwhelmed internally, and then there was a lot of moments, oh, we trained this small, we spent a lot of money on it, and it's kind of like, you know, pretty dumb, at least, like, you know, we have GPT-IV, GPT-III alrea…”
Tworek: Traditional human data labeling is becoming obsolete as models advance
“I think, like, in a way, I think it's getting more and more to be a thing of the past as the models are getting smarter and smarter. This is becoming less of a thing, but I think a few years back, and especially in GPT-IV days, this was the thing.”
Tworek: Pre-Training on Unlabeled Data Yields Far More Intelligence Than Supervised Mapping
“There are many more bits usually in the targets than in the labels and studying the structure of targets itself. It yields much more learning and much more intelligence than learning the mapping itself. So like spending a whole compute on just learning the dat…”
Tworek: OpenAI's o1 release caught US AI labs unprepared for RL
“As far as I know, like our O-one release mostly caught a lot of us labs by surprise. They didn't have like similarly advanced RL research program to my knowledge, basically no one.”
Tworek: OpenAI's RL algorithm is not GRPO but shares similar components
“Like what we, what OpenAI is doing is not exactly GRPO. It is slightly different in many different ways, but like some parts are definitely similar.”
Jerry Tworek: Pre-training AI models is mathematically simple compared to RL
“The first thing that is important to know and understand, RL is hard. Like, conceptually, if you think about it, and there's still a lot of depth to it, but very conceptually, mathematically speaking, pre-training is dead simple.”
Tworek: OpenAI internal models can currently reason for hours on tasks
“The models I can think for, like, 30 minutes, hour, two hours these days on certain, certain types of Tasks and problems like even, even, even longer than that.”
Tworek: OpenAI does not currently train models online via live interactions
“This is not what I am aware, at least, like, not, not, not what OpenAI is doing at the moment”
Tworek: Online RL shouldn't be used at ChatGPT's scale without robust safeguards
“So I, at least until, until we have a really good safeguards, I don't think we should try to do that in anything like as complex and large scale as ChatGPT.”
Tworek: AI models need deep understanding of consequences for alignment
“I don't think like you can just tell them all like, A few show with a few good things to do, and it will do them all needs to deeply understand its action and consequences to really be able to choose the right thing.”
Tworek: AI alignment is a never-ending pursuit as human goals evolve
“And it's I think it's a never ending pursuit because like, even, even for humans, it's not super easy to define what's, what do we consider a light? And I think as our civilization will evolve, it will, the notion of alignment and the goals of humanity will, K…”
Tworek: OpenAI's competitive programming performance was a research byproduct
“I think we used like specifically programming puzzles for a while as a, Very nice research test bed of our ideas. Those are nice problems to experiment on them, and they weren't ever, like, considered part of the product, but it's, those are pretty, like, comp…”
Tworek: OpenAI placed second in AtCoder behind a former employee
“We did also like IOI, International Olympic Informatics earlier this year, and Adcoder Heuristics competition as well, where we went second, we, behind a single human that is also a Polish person that used to be employed by OpenAI some time ago.”
Tworek: Reward hacking in AI mirrors human behavior under flawed incentives
“In some way you can say it's a limitation of reinforcement learning, but when I was thinking about it, I realized a lot of that happens in human systems as well. There are a lot of like incentive system and reward systems and even, even happens in workplaces a…”
Tworek: Pre-training and RL are necessary for AGI, but not sufficient
“I generally think something that we are doing, like, pre-training today is necessary. I think something that, like, we are doing RL today is necessary, and there will surely be a few things more, and like, we have a lot of, Very ambitious research programs on …”