Oct 2, 2025 · 1h 10m · mad
Sonnet 4.5 & the AI Plateau Myth — Sholto Douglas (Anthropic)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The MAD Podcast, host Matt Turck interviews Anthropic AI researcher Sholto Douglas about the release of Claude Sonnet 4.5, frontier model scaling paradigms, and why claims of an AI plateau are premature. Douglas shares insights on reinforcement learning, 30-hour autonomous coding agents, and his personal journey from competitive fencing to leading infrastructure research.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 16.8% of the talking time here. How this is scored →
Matt as informed peer 3.9 Guest teaching 4.0 Guest disagreement 0.6 Matt pushing back 0.9
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:02:26 Debunking the AI plateau myth with the sailboat analogy
Sholto aggressively dismisses the popular AI plateau narrative, arguing that current LLM pipelines are primitive duct-tape systems with massive room for expansion.
Hardest push from Matt ▶ 59:38 Challenging the lab consensus with Yann LeCun's counter-thesisMatt directly pushes back against lab optimism by citing prominent skeptics like Yann LeCun who claim current architectures cannot reach AGI.
Biggest teaching moment ▶ 48:08 Textbook skimming versus workbook problem solvingSholto offers a crystal-clear masterclass analogy explaining why reinforcement learning unlocks capabilities that pre-training alone can never produce.
Matt holds his own ▶ 21:07 Defining Richard Sutton's Bitter LessonMatt demonstrates deep familiarity with foundational AI literature by interjecting to cite and explain Richard Sutton's Bitter Lesson essay.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Rapid Model Releases and Hardware Lead Times | 3 | 3 | 1 | 1 | Matt asks about Anthropic's rapid release pacing and model hierarchy. Sholto explains the two-paradigm regime of pre-training plus RL scaling and compute supply lead times. | |
| Growing Up in Australia and Athletic Mentorship | 1 | 1 | 0 | 0 | Matt asks about Sholto's background in Australia. Sholto describes his mother's medical mentorship and his elite international fencing trajectory. | |
| Fencing as Reinforcement Learning and Discovering AI | 4 | 2 | 0 | 0 | Matt perceptively framing YouTube-based athletic analysis as an early form of reinforcement learning. Sholto agrees and outlines reading Gwern's scaling essay and starting independent robotics research. | |
| Academic Obstacles versus Independent High-Signal Work | 4 | 5 | 1 | 1 | Matt probes the difference between academic gatekeeping and AI research effectiveness. Sholto explains how independent artifacts like Simon Boehm's CUDA guide provide higher signal than traditional PhD credentials. | |
| Navigating Gemini's Formation and Building Inference Stacks | 2 | 4 | 0 | 0 | Sholto details joining Google before ChatGPT, building Gemini's inference stack from scratch to save hundreds of millions, and navigating multi-team sociopolitical dynamics. | |
| Transition to Anthropic and Shared Alignment Focus | 3 | 4 | 1 | 1 | Matt asks how top AI labs differ given similar compute resources. Sholto contrasts DeepMind's scientific focus with Anthropic's laser focus on alignment and near-term economic impact through coding. | |
| Mechanistic Understanding and Sutton's Bitter Lesson | 6 | 4 | 0 | 1 | Matt demonstrates explicit AI research knowledge by interjecting to define Richard Sutton's Bitter Lesson essay. Sholto explains how researcher taste acts as a simplicity regularizer in single training runs. | |
| Convolutional Networks, Vision Transformers, and Language Structure | 2 | 5 | 0 | 0 | Sholto provides concrete architectural examples of Sutton's Bitter Lesson, explaining why CNN priors and explicit grammar structures get washed away by general vision transformers and language scale. | |
| Balancing Short-Term Wins with Long-Term Techniques | 4 | 4 | 0 | 1 | Matt asks about research failure rates and managing expensive compute budgets. Sholto quotes Noam Shazeer's 10 percent idea hit rate and discusses balancing short-term tweaks with fundamental long-term research. | |
| Anthropic's Targeted AGI Bet versus DeepMind's Broad Scope | 4 | 5 | 1 | 1 | Matt asks whether Anthropic researchers investigate non-transformer or non-RL paradigms. Sholto explains Anthropic's focused AGI bet versus DeepMind's broader exploration, highlighting coding's unique verification tractability. | |
| Benchmark Metrics, SWE-Bench Saturation, and Real-World Impact | 4 | 4 | 1 | 1 | Matt requests hard facts and metrics on Sonnet 4.5's SWE-bench score. Sholto cites the jump from 72 to 78 percent, notes benchmark saturation, and emphasizes real-world partner adoption like Cognition's Devin. | |
| From Short Prompts to 30-Hour Autonomous Execution | 5 | 3 | 0 | 1 | Matt articulates a sharp startup thesis about betting on six-month model capabilities. Sholto agrees enthusiastically and highlights Sonnet 4.5's ability to run autonomously for 30 hours to build full applications. | |
| Terminal Loops, Memory Management, and Self-Correction | 4 | 5 | 0 | 1 | Matt asks what an agent physically does during 30 hours of execution. Sholto breaks down terminal execution loops, tool use, markdown memory files, and emergent self-correction capabilities. | |
| End-to-End Application Generation and Replicating Claude.ai | 5 | 3 | 0 | 0 | Matt brings up Anthropic's viral showcase demo where Claude autonomously replicated Claude.ai with artifacts. Sholto notes this represents the first steps toward fully functional application synthesis. | |
| Overcoming Context Limits and Teaching Model Taste | 4 | 5 | 0 | 1 | Matt pushes on the underlying technical differences between 7-hour and 30-hour agent limits. Sholto discusses Tesla-style intervention rates, global context drift, and multi-agent systems for teaching coding taste. | |
| Cumulative AI Progress and Moore's Law Analogies | 3 | 5 | 0 | 1 | Matt asks about the core technical breakthroughs behind Sonnet 4.5. Sholto provides an intuitive analogy explaining pre-training as skim reading textbooks and RL as working out problems with graded feedback. | |
| Overlap of Test-Time Compute and Reinforcement Learning | 5 | 5 | 0 | 1 | Matt raises historical RL achievements like AlphaGo and Sutton's decades of research to ask why LLM RL converged in 2025. Sholto explains that simple reward feedback on strong base models beat complex search trees. | |
| Defining AGI and Evaluating Future AI Capabilities | 4 | 5 | 1 | 2 | Matt playfully pushes back on Sholto calling a harder AGI criteria stronger. Sholto defines AGI as outperforming most humans on computer tasks within 2-3 years and discusses parallel execution leverage. | |
| Addressing Counter-Theses and Architectural Debates in AI | 6 | 4 | 2 | 3 | Matt directly confronts Sholto with Yann LeCun's counter-thesis that transformers and current paradigms are fundamentally flawed. Sholto rejects the premise, emphasizing consistent benchmark progress. | |
| Debunking the AI Plateau Myth and Economic Evaluations | 5 | 4 | 3 | 1 | Sholto forcefully debunks the AI plateau narrative, comparing LLM pipelines to duct tape versus centuries-old sailboat design. Matt adds details about economic GDP evaluations. | |
| Preparing for Individual Leverage in an AI-Driven World | 4 | 5 | 1 | 1 | Matt asks how individuals should prepare and raises physical hand manipulation limits in robotics. Sholto debunks Moravec's paradox as a data artifact and explains generator-verifier gaps in robotics. |
Statements from this episode (50)
Douglas: AI releases accelerating due to dual-paradigm scaling
“There's now this two paradigm regime where previously you did free training scaling and reinforcement learning scaling, and now we're in a mix of the two basically. And so I think that gives you more opportunities to update models because it means that you can…”
Douglas: TSMC bottlenecked desired AI chip acquisitions in 2024
“Even if you, as much as you wanted chips last year, it would have been impossible to get them because TSMC was, you know, booked out and so forth.”
Douglas: The post-ChatGPT AI compute supercycle begins properly in 2025
“So finally, this year is where the compute, like, super cycle is, like, beginning properly in effect.”
Douglas: Anthropic's mid-tier Sonnet is smarter than its flagship Opus
“One of the interesting things about this most recent release is actually Sonnet is smarter than Opus.”
Douglas: Reinforcement learning elevates mid-tier models to match older flagships
“So that allows you to take a mid-tier model and make it as good as a larger-tier model of six months ago or three months ago.”
Douglas: Independent technical blogs are the highest AI hiring signals
“The fastest route, or like, the most immediate one is whenever we see a really good blog post where people have, like, done incredible amount of work in an independent fashion, it's one of the highest signal things there is.”
Douglas: Google LLM inference stack saved hundreds of millions in months
“This ended up saving several hundred million dollars, I think, like, even over the first six months”
Douglas predicts DeepMind will lead the world in AI science discoveries
“DeepMind, if you wanted to solve science, is the best place in the world. Like, I think that DeepMind will directly contribute to more scientific discoveries from AI than anything else, right?”
Douglas: Anthropic focuses on alignment and near-term economic coding utility
“Anthropic has been laser focused on on two things. One is, like, long-term AI alignment, and two is near-term economic impact. So Anthropik has been laser-focused on coding and computer use, and things that we think will make a direct impact to the economy, li…”
Douglas: Anthropic intentionally deprioritized math reasoning unlike OpenAI and DeepMind
“You know, one thing that Anthropik, like, You noticeably hasn't focused on compared to DeepMind and to OpenAI is is mathematical reasoning, right? DeepMind and OpenAI have been pursuing mathematical reasoning because of the implications for science and for sci…”
Douglas: Compute scale consistently wipes out clever human priors in AI
“Generations of people have developed clever methods of encoding priors about how they think an artificial intelligence should reason. And encoding it into the model, and all of this gets wiped out by scale and, you know, planning, like, basically, like, search…”
Douglas: Vision transformers outscale CNNs by leveraging global image context
“Convolutional neural networks will be better than a more general, like, a vision transformer for the vast majority of images up to a certain, and up to a certain amount of scale. But past a point, actually, you need to be able to flexibly integrate information…”
Douglas: Even AI pioneer Noam Shazeer only sees 10% of ideas work
“I once asked this question of Noam Chazier. And he was like, yeah, maybe like 10% of my ideas work, and that's not, right? You know, one of the, you know, an absolute genius, one of the best in the field. So if only 10% of his ideas work, then I think that, yo…”
Douglas: Top AI labs give researchers months to explore unproven ideas
“It's one of the things that at both Anthropic and at DeepMind we really tried to build, which is like a culture of safe experimentation where People were trusted to explore ideas for a long time out in the wild. Cause you often need months of independent resea…”
Douglas: Examining training data and simple tweaks yields massive model gains
“A really high ROI use on, on your time would probably be to just go and look at the data and think hard about what, like the model is learning or doing and like make some tweaks that makes it like, there's so many, like you could even just The simplest things …”
Douglas: Anthropic believes AGI is reachable in a couple of years
“We think that, you know, AGI is within reach in the next couple of years.”
Douglas: Anthropic's ethos is that scaling current techniques achieves AGI
“Like really for the last five or six years, Anthropics ethos has been scaling compute with broadly the current set of techniques is like AGI is tractable within those bounds.”
Douglas: DeepMind has 1,000 on Gemini and 10,000 on foundational research
“But if you look, Gemini is like, you know, a thousand people, there's a, there's still like 10,000 plus people doing all kinds of really like longterm foundational research at DMI.”
Douglas: AI takeoff speed depends on AI assisting AI research
“We think that one of the most important signals of whether or not we are basically the speed of takeoff, the speed of progress is driven by how much AI is able to assist AI research.”
Douglas: Coding is uniquely tractable for AI because execution is verifiable
“Coding is a uniquely tractable problem in some respects for the techniques that we have in terms, the data exists in many ways. You can containerize and run things in parallel. You can run unit tests, and so you can verify.”
Douglas: Sonnet 4.5 pushed SWE-bench scores from roughly 72% to 78%
“We moved recently from roughly 72 to roughly 78 in Sweepbench”
Douglas: The entire AI industry scored under 20% on SWE-bench last year
“As recently as a year ago, I think we were under 20% or something like that as a field.”
Douglas: Cognition rebuilt Devin's architecture around Anthropic's highly useful Claude Sonnet
“The Cognition folks in Devon found the model so useful, they had to, like, rebuild their architecture around it.”
Douglas: Claude 3.5 Sonnet drove product-market fit for code editor Cursor
“In many ways, this model is what caused PMF for Cursor. Cursor took off like a rocket, right, with because they were in the right place, and they were able to capitalize on that model as offering a coding experience that didn't previously exist.”
Turck: AI startups in 2025 must build for capabilities six months away
“One of the key lessons for anybody in the startup world. In 2025, which is bet on what the models will be able to do in six months from now.”
Douglas: AI coding agent supervision will drop to 20-minute intervals within months
“Over time you know, over the next couple of months, you're probably gonna end up in a situation where you only need to supervise the models every, you know, 10 minutes, 20 minutes or so.”
Douglas: An Anthropic AI agent operated autonomously for 30 hours building apps
“We asked it to build something that looks roughly like a chat app, you know, something like Slack or, you know. And it was, it, the model just worked for 30 hours. Like, it was just spinning there on a computer for 30 hours, and came out with a really good wor…”
Douglas: Anthropic built persistent memory files into its AI agentic harness
“One of the things that we're pretty happy with about the recent launches, we've finally taught the models to use what's called memory. And so, and we've built that into the agentic harness. So it's able to create a markdown file of to-dos and things that it th…”
Douglas: The current generation of AI agents are astonishingly good at self-correcting
“I think one of the things that made me remarkable about the current generation of agents is that they can self-correct. In fact, they're astonishingly good at self-correcting.”
Douglas: AI autonomous task execution time horizons double every six months
“And so I think it's like every couple of months, the time horizon that the AIs are capable of doing is doubling or something, something crazy. Maybe, maybe every six months the time horizon doubles”
Douglas: Anthropic models autonomously replicated the Claude.ai website in hours
“And in this case, the model replicated Claude.ai with artifacts, with everything else I can't quite remember how long that one took. Maybe a couple hours to do.”
Douglas: AI application development will see another massive leap next year
“Over the next six months, over the next year, expect dramatic progress here. And like look at where we are now versus where we were a year ago. And the difference is I expect the same jump basically.”
Douglas: AI coding interventions stem from taste, not raw programming capability
“Right now you need to intervene quite frequently, but it's usually on questions of taste rather than it is questions of, like, raw programming ability.”
Douglas: AI progress on METR evaluations plots as a straight line
“Like on the meter eval, if you look at progress over the last two years, you can plot it with a straight line, right?”
Douglas: AI progress relies on accumulated work and compute, not single breakthroughs
“It's not any one critical breakthrough. It's more the accumulation of a huge amount of work in an environment where there's a sort of exogenous force of compute pushing progress forward.”
Douglas: Solving AI hallucinations intrinsically requires reinforcement learning
“Saying, I don't know, or solving, you know, hallucinations is, ah, intrinsically requires reinforcement learning in many ways.”
Douglas: Reinforcement learning on language models finally started working in late 2024
“I think also an important change in, you know, in this sort of like era of reasoning models and RL on language models is, at the end of last year, RL on language models finally started to work.”
Douglas: OpenAI's o1 established test-time compute and RL as a scaling axis
“And I think OpenAI deserves a lot of credit for you know, releasing the first, like, serious RL plus LLMs release with O-one. And I think this really kicked off a pretty, you know, substantial change because it opened up a new axis of scaling, right? There was…”
Douglas: Test-time compute executes reasoning while RL provides feedback on correctness
“One way of thinking about this is test time compute is doing a lot of reasoning, and then RL is the feedback signal on whether or not that reasoning was right or wrong.”
Douglas: Test-time compute solves harder tasks before RL distills them into models
“Test time compute lets you do harder problems than you can currently do, than you can like currently do off the cuff, and RL
Then allows you to sort of distill that back into the model.”
Douglas: Simple RL methods work better on language models than complex strategies
“One of the craziest things about RL on language models in the, in, like, the RL from verified rewards regime, is it's almost the simplest possible thing. It's, like, almost too simple to work. And this is, again, comes back to that question of taste, where Rea…”
Douglas: AI reasoning strategies emerge naturally with enough compute and RL feedback
“Give it math questions, tell it whether it got them right or wrong, and the model will learn. This is, it comes down to a bit of lesson in scale and search, is just allow the model to search, have enough compute to run the experiments, and the model actually e…”
Douglas: AI industry will reach human-level computer capabilities in 2-3 years
“Which is that in the next two or three years, given the right feedback loops, given the right compute, given the right, you know, elbow grease and this kind of stuff, we think that we as the AI industry are all on track to create something that is at least as …”
Douglas: Transformers successfully model any domain given sufficient data and compute
“I don't think that's true. I think we haven't yet really found anything that transformers haven't been able to model provided sufficient data and sufficient compute.”
Douglas: LLM pipelines are two and a half years of desperate effort
“When I look at an LLM training pipeline, it is two and a half years of best effort, last minute, desperate effort. And there's just so much room to go on every part of it.”
Douglas: Anthropic's Claude Opus 4.1 led OpenAI's GDP eval benchmark
“4.1 Opus was the leading model there.”
Douglas: AI beating GDP benchmarks won't immediately alter the broader economy
“We'll probably reach like better than human on the GDP eval, and it won't change anything economically because It'll be all the connective tissue, and all the, like, you know, the context, and actually, like, the task won't be representative.”
Douglas: Individuals will manage 24/7 AI agent teams within two years
“If coding agents progress in the way I've been saying, in a year or two, you'll be able to manage a team, basically, that works 24 seven for you doing work.”