Feb 27, 2026 · 1h 5m · latent-space
Measuring Exponential Trends Rising (in AI) — Joel Becker, METR
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space podcast, METR's Joel Becker discusses empirical AI safety evaluations, the methodology behind the Model Time Horizon benchmark, the dynamics of automated AI R&D, and his personal strategies in prediction markets.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 39.2% of the talking time here. How this is scored →
The hosts as informed peer 5.6 Guest teaching 5.0 Guest disagreement 1.9 The hosts pushing back 3.1
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 13:59 Direct rejection of forecasting credentials and discontinuous progress
Joel explicitly challenges Swyx's framing by stating he wants to attack two claims, denying he was ever a formal superforecaster and disputing that Opus 4.5 represented an unprecedented break in the trendline.
Hardest push from the hosts ▶ 58:37 Alessio and Swyx defend pragmatic scaffolding over pure wait-and-seeAlessio and Swyx challenge the fatalistic view that all scaffolding is useless because future models will replace it, insisting developers must build custom harness optimizations for immediate production needs.
Biggest teaching moment ▶ 19:17 Joel explains systemic biases in developer productivity metricsJoel methodically breaks down why anecdotal and self-reported 10x speedups are economically misleading, explaining how concurrency, task selection bias, and low marginal value distort uplift calculations.
The host holds their own ▶ 39:29 Swyx details cluster timelines and unobserved hardware capexSwyx demonstrates deep hardware and industry supply-chain knowledge, noting how GPU deployment schedules at major labs mathematically dictate model release windows beyond public financial disclosures.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Distinguishing Model Evaluation from Threat Research | 4 | 4 | 1 | 2 | Swyx and Alessio ask clarifying questions regarding METR's balance between model evaluation and threat research, as well as how threat models have evolved. Joel explains the shift from autonomous replication toward automated R&D acceleration. | |
| The Origin and Methodology of the Time Horizon Benchmark | 5 | 5 | 1 | 2 | Alessio probes the origins of the time horizon metric and how METR selects tasks without introducing arbitrary bias. Joel details the internal history, straight trendline emergence, and task constraints like low-context solvability and autograding. | |
| Task Distributions: From SWAs to RE-Bench | 6 | 5 | 2 | 3 | The hosts and guest discuss the differences between SWA, HCAST, and RE-Bench tasks, and why task difficulty is measured in human hours rather than agent wall-clock execution time. Joel and Swyx agree on dismissing sensational, unscientific agentic claims. | |
| Opus 4.5 and the Trend Line Trajectory | 6 | 6 | 4 | 4 | Swyx suggests Opus 4.5 broke METR's trendline and references Joel's forecasting background. Joel directly rejects both premises, clarifying he is not formally a superforecaster and arguing model progress remains remarkably continuous across compute scales. | |
| Evaluating Developer Productivity and RCT Methodologies | 6 | 6 | 2 | 4 | Alessio and Swyx discuss developer productivity RCTs and empirical workflows. Joel educates on methodological hazards in new studies, including developer selection bias, concurrent task multitasking, and marginal value distortion. | |
| Capability Explosions, Automated R&D, and Physical Loops | 6 | 5 | 2 | 4 | Swyx and Alessio question whether capability explosions will follow continuous curves or sudden phase transitions like boiling water. Joel analyzes the conditions for recursive self-improvement and notes that hardware bottlenecks prevent immediate software explosion. | |
| The Long Tail of AI R&D Automation and Metric Nuance | 6 | 6 | 3 | 4 | Swyx critiques the reduction of AI capability tracking to single summary numbers and advocates for multidimensional capability lists. Joel counters by highlighting the vast real-world physical tail required to automate R&D and challenges Swyx to enumerate an exhaustive list. | |
| Compute Bottlenecks and Algorithmic Progress | 7 | 5 | 1 | 3 | Alessio and Joel analyze how compute slowdowns could directly halve algorithmic progress. Swyx demonstrates strong industry domain knowledge, citing cluster timelines, unobserved hyperscaler capex, and SemiAnalysis insights. | |
| Prediction Markets, Manifold Alpha, and Market Ethics | 6 | 6 | 3 | 4 | Alessio and Swyx ask Joel about ranking #1 on Manifold Markets. Joel explains his arbitrage maneuver on charity matching markets and questions the broader societal value of prediction markets given gambling dynamics, while Swyx defends price discovery. | |
| Frontier Evals: AI Village and Real-World Transcripts | 6 | 6 | 1 | 3 | Joel outlines cutting-edge evaluation approaches such as AI Village and raw transcript analysis. Swyx links these ideas to Noam Brown's multi-agent cooperation research and DeepMind's open-endedness teams. | |
| Scaffolding, Harness Optimization, and Skill Depreciation | 7 | 5 | 2 | 5 | Alessio brings up Terminal Bench harness variations and argues in favor of aggressive task overfitting for immediate developer utility. Swyx and Joel debate whether custom scaffolding creates lasting value or gets washed away by base model advancements. | |
| METR's Roadmap and Hiring Priorities | 5 | 4 | 1 | 2 | Swyx asks about METR's roadmap toward 2026/2030 and what specific qualities METR seeks when rejecting average candidates. Joel outlines requirements around rigorous data intuition, transparent scientific writing, and scrappy execution. | |
| Live Band Karaoke and Parting Reflections | 3 | 2 | 1 | 1 | The conversation closes with lighthearted personal discussion about live band karaoke, acapella music history, and the enduring human element of live performance compared to synthetic AI generation. |
Statements from this episode (21)
Becker: Current Frontier Models Cannot Cause Catastrophic Harm
“We find, we think it's not capable enough, you know, on the basis of some of this capabilities evidence that you've alluded to commit these catastrophic harms.”
METR Shifts Focus from Autonomous Replication to AI R&D Acceleration
“So, something like the autonomous replication threat model, that is being able to set yourself up and control resources, something like that, has been deprioritized relative to AR and D acceleration. That is, you know, the possibility there could be some capab…”
Becker: AI Models Lag on Time Horizons for Vision Tasks
“Tasks that are requiring of vision capabilities, they're probably to take one example, they're probably much less capable today as measured by time horizon, as for these tasks that are typically not requiring vision, vision capabilities that we give them.”
Becker: Low-Context Benchmark Criteria Exclude Real-World Situational Work
“Could a low-context human who was sufficiently skilled at sort of The general skills, but maybe, maybe not the particulars in the background would they be able to achieve success on, on this task? And I think that, that rules out a lot of real work because, yo…”
Becker: Claude Opus 4 Solves Atomic Software Tasks 100% Reliably
“Opus-IV. I'm sure can do that task a hundred percent of the time.”
Becker: METR's Hardest Benchmarks Require 20 to 30 Hours of Autonomy
“Then we go up to HCOS tasks, which span from, you know, only a little harder than those small tasks, all the way up to, you know, something like 20:30 hours, which are requiring of more autonomy, more sort of more sort of sequential actions.”
Becker: Time Horizon Metric Measures AI Task Difficulty in Human Time
“You know, instead we're just plotting what's the difficulty of tasks they can do over time, and that difficulty is measured in human time.”
Swyx: Public AI Agent Performance Claims Are Unscientific and Marketing-Driven
“The state of people making claims on agent performance is very unscientific and much more anecdotal and sometimes influenced by marketing desires.”
Becker: AI Progress Remains Highly Continuous Across Compute Scales
“You know, in some ways, I think the story of Time Horizon is that progress has been remarkably continuous over, over so many years, so many orders of magnitude of compute and effective compute.”
Becker: METR Is Rerunning Its Developer Productivity Randomized Controlled Trial
“We have been redoing it in the background.”
Becker: Overly Bullish AI Developer Speedup Estimates Are Inflated
“I do think that very bullish estimates of speed up today are, you know, to some extent inflated by what we document in that original paper, that people's expectations of speed up tend to be too optimistic, it seems. They also tend to be inflated, I think, by n…”
Becker: Operational Long Tail Will Delay Full AI R&D Automation
“There's this, Very long tail of things potentially involved in in R&D that would perhaps need to be fully automated in order to lead to capabilities explosion. I expect we're measuring, you know, in some ways only, only a small proportion of, only a small prop…”
Becker: Algorithmic Progress in AI Is Strictly a Function of Compute
“The suggestion in this paper is that if you think that algorithmic progress, you know, that, that is coming up with the transformer, coming up with RLHF, you know, MOEs, all of this stuff, better learning rate schedules is, is is itself a function of compute b…”
Becker: Halving AI Compute Growth Halves Algorithmic Progress and Milestones
“And both of them both of those components half when compute halves sort of trivially, because compute is halving, and algorithmic progress halves because compute is this important input, and compute halves, then you might expect time horizon growth to half. An…”
Becker: Donated $5,000 Won on Manifold Markets to Charity
“And then I ended up donating, I can't remember exactly how, how much it was, not, not so much, something like 5000 dollars .”
Becker: Prediction Markets' Social Value May Not Justify Gambling Harms
“I think gambling like behaviors are socially costly and the value of higher quality information is is, is, is real, but, you know, is it worth that disbenefit of people trading away their money, it's not, you know, it's not so clear to me.”
Swyx: Latent Space Holds Embargoed AI News Without Trading
“We do work with people on embargo and we don't trade.”
Becker: AI Coding Fails at Merge Readiness Despite High SWE-bench Scores
“Maybe one that I'll call out there is this difference between whether models pass unit tests, whether they succeed by, you know, SWE bench-like scoring kind of meter-like scoring, benchmark-style scoring, versus whether their solution would be merged into main…”
Becker: AI Scaffolding Value Does Not Persist Across Model Generations
“Within model generation, it's valuable, and across model generations, it's not so valuable.”