Becker: AI Coding Fails at Merge Readiness Despite High SWE-bench Scores
Joel Becker · Measuring Exponential Trends Rising (in AI) — Joel Becker, METR · Feb 27, 2026 · at 52:26
Joel Becker of METR discusses limitations in current coding evaluations with hosts Alessio Fanelli and Swyx on the Latent Space podcast.
0:00 / 0:36exact quote · 36.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Maybe one that I'll call out there is this difference between whether models pass unit tests, whether they succeed by, you know, SWE bench-like scoring kind of meter-like scoring, benchmark-style scoring, versus whether their solution would be merged into main. That is, you know, whether the solution adds tests where it should or doesn't. Whether it follows existing patterns in the code base, whether it makes sure that, that its changes sort of speak to other parts of the code base in, in, in, in appropriate ways. That, that seems very interesting to me. You know, I think model capabilities probably are lagging behind there somewhat versus versus that, which you might see on sweet bench like scoring.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Joel Becker
Becker: Overly Bullish AI Developer Speedup Estimates Are Inflated
“I do think that very bullish estimates of speed up today are, you know, to some extent inflated by what we document in that original paper, that people's expectations of speed up tend to be too optimistic, it seems. They also tend to be inflated, I think, by n…”
Becker: Operational Long Tail Will Delay Full AI R&D Automation
“There's this, Very long tail of things potentially involved in in R&D that would perhaps need to be fully automated in order to lead to capabilities explosion. I expect we're measuring, you know, in some ways only, only a small proportion of, only a small prop…”
Becker: Algorithmic Progress in AI Is Strictly a Function of Compute
“The suggestion in this paper is that if you think that algorithmic progress, you know, that, that is coming up with the transformer, coming up with RLHF, you know, MOEs, all of this stuff, better learning rate schedules is, is is itself a function of compute b…”
Becker: Halving AI Compute Growth Halves Algorithmic Progress and Milestones
“And both of them both of those components half when compute halves sort of trivially, because compute is halving, and algorithmic progress halves because compute is this important input, and compute halves, then you might expect time horizon growth to half. An…”