Jun 25, 2026 · 41m · latent-space
Cooking with OpenAI’s Research Chief: AGI, o1, Evals, and Scaling Laws — Mark Chen
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space cooking series, OpenAI Chief Research Officer Mark Chen joins host Alan to cook Korean tofu stew and flambéed prawns while discussing the genesis of the o1 reasoning model, scaling laws, benchmark evaluation crises, and the path to autonomous AGI research.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 3.6 Guest teaching 5.1 Guest disagreement 1.2 The hosts pushing back 1.1
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 9:13 Firm rejection of scaling law skepticism
Mark strongly disagrees with prevailing bearish takes on pre-training, noting that bottleneck claims have consistently proven false across ten orders of magnitude.
Hardest push from the hosts ▶ 25:54 Challenging naive context expansionAlan challenges the simplistic low-hanging fruit solution of merely increasing context windows, arguing it introduces context bloat and rot.
Biggest teaching moment ▶ 22:15 Architecting adversarial eval teamsMark explains the structural necessity of separating evaluation teams from model training teams to eliminate perverse incentives and benchmark gaming.
The host holds their own ▶ 17:42 Contrasting engineering vs research lifecycleAlan articulates the distinct transition paths of early ideas to end-user products between production engineers and frontier researchers.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| From Trading to AI: Developing Research Taste | 3 | 5 | 1 | 1 | Alan prompts Mark on how trading experience translates to AI research and how non-PhDs build research taste. Mark breaks down how replication of landmark papers like ResNet and PixelCNN is the practical key to building research instincts rather than relying on pedigree. | |
| RL Horizons and Evaluating Superhuman Capabilities | 4 | 5 | 1 | 1 | Alan asks about reinforcement learning boundaries across subjective domains and evaluating superhuman benchmarks beyond IMO. Mark explains that RL struggles where grading is subjective (like creative writing) versus objective ground truths (like math and coding). | |
| Scaling Laws and the Genesis of OpenAI o1 | 3 | 6 | 2 | 1 | Alan asks Mark about common contrarian takes like 'pre-training is dead.' Mark firmly rejects bearish views on scaling laws, explaining how recurring historical bottlenecks were consistently overcome and describing the early internal conviction needed to spawn o1. | |
| Research Leadership and Roadmap Prioritization | 3 | 4 | 1 | 1 | Alan inquires about OpenAI's unchanged research roadmap and how leadership balances top-down steering with bottom-up researcher ideas. Mark describes OpenAI's meritocratic management culture and how compute allocation checkpoints force periodic re-evaluations. | |
| Compute Allocation and Identifying Top Research Talent | 4 | 5 | 1 | 1 | Alan asks how OpenAI sifts through hundreds of research proposals and identifies standout talent. Mark explains directive compute allocation where managers receive dedicated large pools alongside flexible discretion to support diverse researcher profiles. | |
| Navigating the Evals Crisis and Adversarial Benchmarking | 4 | 6 | 1 | 1 | Alan raises the discrepancy between benchmark scores and vibe checks. Mark details the industry's evals crisis due to benchmark saturation and advocates for strictly separating eval design teams from model optimization teams to maintain an adversarial dynamic. | |
| Jakub Pachocki Dynamics, Jagged Frontiers, and Context | 4 | 5 | 1 | 2 | Alan pushes on why models excel at complex Olympiad problems but struggle with basic human tasks, citing context bloat and context rot. Mark discusses the jagged frontier of intelligence and contrasts naive context window expansion with state compaction techniques. | |
| Flambéing Prawns, AGI Horizons, and Autonomous Research | 3 | 5 | 2 | 1 | Alan asks whether AGI requires multiple drastic paradigm breakthroughs such as continual learning. Mark gently pushes back on that framing, arguing that continual learning is an approachable primitive with multiple viable shots on goal rather than an insurmountable barrier. | |
| Multimodal Architectures, Vibe Researchers, and High-Risk Bets | 4 | 5 | 1 | 1 | Alan probes unified multimodal architectures, vibe researching, and managing researchers whose bets repeatedly fail. Mark details why shared infrastructure stacks are favored and how a high-risk portfolio philosophy accommodates stringed failures before a breakthrough. |
Statements from this episode (16)
Mark Chen brought soup to researchers to counter Zuckerberg's talent poaching
“Oh, you know, it's absolutely a true story. And I have brought soup to our own researchers.”
Chen: Meta poaching has calmed down and OpenAI came out on top
“I think that met us calmed down a little bit. I think we came out on top”
Mark Chen: A PhD is not necessary to excel in AI research
“There are a lot of researchers who just started out without formal training in machine learning or AI research. We've very much believed in training people up to do this. I think the real hard thing is the ability to creatively solve problems and think outside…”
Mark Chen: Paper replication is the best way to develop AI research taste
“The best mechanism I've found for developing that is really just replication. So I think you should take papers that you really look up to, and just try to fully replicate it.”
Mark Chen: AI models are producing 'Move 37' breakthroughs in math and coding
“There's move-thirty-sevens in, in math. There's in computer science and coding. I think even, yeah, just, it feels like a lot of people woke up at the start of this year and were like, man, agents are working in my profession. And you know, they're essentially…”
OpenAI's Chen: Reinforcement learning struggles in subjective, hard-to-grade fields
“RLs traditionally had headwinds when it's come to fields that, you know, it's more kind of, Subjective than objective. So if you kind of think of, you know, one kind of, you know example of this is creative writing, where, you know, you could take two pieces o…”
OpenAI's Chen: AI models already discover novel theorems and advance sciences
“The initial direction we took was you should move it to real world research, right? And we've seen that the models, they've gotten a lot better at just kind of discovering novel theorems and pushing the frontiers of hard sciences. Even today, right, that's no …”
Mark Chen: AI scaling laws will continue to hold
“And so I think it's just more and more of the same, right? Like more careful research engineering, more careful data engineering, more careful scaling, and it always unlocks that next ability to scale further. So I mean, it's held for You know, almost 10 order…”
Jakob Pachocki and Ilya Sutskever overcame internal inertia to build OpenAI o1
“Even at a company like OpenAI, you would have people ask naturally, why do something when you have a machine that works? And fundamentally, you know, it's to the credit of, you know, Jakob, Ilya, many of the people who really had conviction and vision in this …”
OpenAI's three research pillars are pre-training, RL, and alignment
“At the very highest level, right, we have an org that focuses on pre-training, right, which is, you know, giving models a lot of world knowledge. We focus on RL, like, teaching the models how to reason with that knowledge, how to chain the little insights toge…”
OpenAI's Mark Chen: AI is in an evals crisis with saturated benchmarks
“So I think beyond that, the other scary thing in the field is the number of canonical gold standard benchmarks is low. And we really are kind of in an evals crisis, right? Where all the really great evals that we all know, like growing up, like taking the SAT …”
Chen: Eval teams and model optimization teams must remain separate
“Yeah, I think there's a kind of interesting philosophy of separate the teams that are creating the evals from the teams that are optimizing the models themselves, because that way you don't, like, co-incentivize them, right? Like, the way the evals theme can w…”
Chen: Compaction shortcuts expensive native long context in coding products
“Many, many coding products today have features like compaction, right? Where you can compress kind of either insights or working state and stuff like that, you know, it just shortcuts a lot of the very brutally difficult and expensive permit is that you have t…”
Chen: OpenAI favors unifying modalities in as few architectures as possible
“For a research lab, I think there are a lot of advantages for it to being under one. So you just have to maintain one infrastructure stack, for instance. I think the cost to, like, maintaining and scaling many infrastructure stacks at once I think that's somet…”
Chen: OpenAI's three-year goal is models conducting end-to-end research
“When we look at our kind of three-year roadmap, right the end goal that we want to reach is one where You know, the models are just doing end-to-end research, and I think a part of that problem is just being able to have the model come up with good taste.”