SWE-bench
includes SWE Bench Verified, SWE Bench Pro, SWE Bench Multimodal, SWE Bench Multilingual, SWE Bench Lite, SWE Bench Full, SWE Bench Light, SWE Bench Live
51 statements across 20 episodes · 16 bullish · 18 bearish · 18 people on the record · first statement Aug 22, 2024 by Alistair Pullen · said 299 times in 44 episodes since 2024 · across every show →
Mentions by year, the whole family
brought up most by Shawn Wang (60), John Yang (14), Jesse Hu (14), Graham Neubig (14), Erik Schluntz (14), Shawn Lewis (13), Alistair Pullen (12), Alessio Fanelli (12)
tap a year for its mentions
2026 38 mentions in 7 episodes 5 per episode
- The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals
- 🔬 "The Most Innovative Diffusion Research Is Happening in Drug Discovery, Not Image Generation"
- Measuring Exponential Trends Rising (in AI) — Joel Becker, METR
- Cooking with OpenAI’s Research Chief: AGI, o1, Evals, and Scaling Laws — Mark Chen
- When AI Agents Run Businesses — Lukas Petersson and Axel Backlund of Andon Labs
- Claude Code for Finance + The Global Memory Shortage: Doug O'Laughlin, SemiAnalysis
- Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah Hill-Smith
- every mention in 2026, scene by scene →
2025 128 mentions in 23 episodes 6 per episode
- [State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang
- Beating OpenAI and Anthropic by Looking At Data: the new #1 on SWE-Bench w/ W&B CTO Shawn Lewis
- The #1 SWE-Bench Verified Agent
- Terminal-Bench: Pushing Claude Code, OpenAI Codex, Factory Droid, et al to the limits
- 2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents
- 🕰️ The Oral History of Windsurf (ft. Varun Mohan, Scott Wu, Jeff Wang, Kevin Hou, Anshul R)
- ChatGPT Codex: The Missing Manual
- GPT 4.1: The New OpenAI Workhorse
- 15 more episodes that year, every mention in 2025 →
2024 133 mentions in 14 episodes 10 per episode
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- [Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu
- Is finetuning GPT4o worth it?
- Best of 2024 in Agents (from #1 on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands)
- Language Agents: From Reasoning to Acting — with Shunyu Yao of OpenAI, Harrison Chase of LangGraph
- Building AGI in Real Time (OpenAI Dev Day 2024)
- Windsurf: The Enterprise AI IDE
- Building AGI with OpenAI's Structured Outputs API
- 6 more episodes that year, every mention in 2024 →
Everything said about SWE-bench, oldest first
Aug 22, 2024 positive
Pullen: SWE-bench is currently the best metric for software engineering agents
“It was actually a very useful tool in building Genie because beforehand it was like, yes, vibe check this thing and see if it's useful. And then all of a sudden you have an actual measure to see like, couldn't it do software engineering? Not the best measure, …”
Aug 22, 2024 negative
Step-by-step prompting on foundational models cannot achieve SOTA SWE-bench performance
“I think it's fairly obvious that with the foundational models, like you can ask them to think by step by step and ask them to plan and stuff, but that isn't enough because if you look at how those models score on these benchmarks and then they're not They're n…”
Aug 22, 2024 negative
Cosine refuses to publish SWE-bench trajectories to prevent competitor model distillation
“For the moment, as a closed source company, like fighting for an edge, we've decided not to publish that information for that exact reason. I don't want someone basically taking my tragedies and then taking a model that's suing them in GA and just distilling i…”
Aug 22, 2024 negative
Training on pull request diffs yields code generators, not software engineers
“What the, very crudely, what the pre-trained models are reading is they're reading those final diffs and they're Emulating that and then being able to output it. Right. But of course it's a super lossy thing, a PR. You have no idea why or how, for the most par…”
Aug 22, 2024 neutral
Aug 22, 2024 neutral
Pass-fail metrics like SWE-bench are too sparse for evaluating AI agents
“Sweet Bench is a pass or fail, right? Like you either have solved the problem or you haven't. That is quite sparse. Like it doesn't give you a huge amount of information because your model could have got a lot of it right. Like looking through when you do a ma…”
Aug 22, 2024 positive
Extracting problem-solving history most determines Cosine Genie's SWE-bench performance
“The way that we Decided we want to try to extract what happened in the past, like as forensically as possible, has been and is currently like one of the main things that we focus all our time on. Because doing that as we're getting as much signal out as possib…”
Aug 22, 2024 negative
Genie's SWE-bench success rate drops to roughly 50% past 60k tokens
“Performance of Jeannie over the length of the context window degrades fairly linearly. So actually, I actually broke it down by probability of solving a SWE bench issue. Given the number of tokens of the context window at 60 K, it's basically .5. So if you go …”
Sep 27, 2024 positive
Oct 4, 2024 positive
Pullen: CoScene's Genie outperforms OpenAI o1 out of the box on SWE-bench
“So it was obviously great to see, like, we still are better than O-one out of the box. You know, even with an older model, and I'm sure that that, that Delta will continue to grow once we're able to train O-one and once we've done more work on our dataset usin…”
Oct 4, 2024 negative
Pullen: SWE-bench is a poor proxy for real-world AI coding competence
“I know Sweebench is, like, the most commonly talked about thing, and honestly, it's a very, it's an amazing project, but one of the things we've learned the most from actually shipping this product to users is, it's a pretty bad proxy at telling us how compete…”
Oct 19, 2024 positive
Oct 19, 2024 neutral
Oct 19, 2024 positive
Jesse Hu: SWE-bench gains over baseline GPT-4 come entirely from agent scaffolding
“The diff between that and something like Devin is all in like, sort of like the agent scaffold or the agent code, right? So that's, what's really exciting about this stuff. It shows off what you can do just from prompting and just from adding tools.”
Oct 19, 2024 negative
Hu: Some SWE-bench Tasks Are Unsolvable Due to Missing Context
“Sometimes it feels like you try to read the issue. And you're just like, okay, even if I was like an Oracle or like some sort of God coder, I couldn't, I would not be able to solve this because there's some context that was left out here. There's something tha…”
Oct 19, 2024 neutral
Hu: Coding agents produce bloated edits unless constrained by brevity priors
“There's something nuanced about this data set in particular where all the edits are super short
and it's like a prior that you can put into your code.
But if you don't have that, then yeah, like agents tend to just keep making obnoxiously long edits”
Oct 19, 2024 neutral
Oct 19, 2024 neutral
Jesse Hu: SWE-Bench trajectory requirements deter commercial agents from submitting
“They actually require you to submit these things called trajectories that are proving what you did and that you're not cheating. And a lot of this ends in controversy. A lot of this ends in people say, well, we don't want to submit at all because if we review …”
Oct 19, 2024 negative
Oct 19, 2024 negative
Hu: SWE-bench repos do not represent typical app development work
“And I'll say the critique here is that these are, like, really robust you know, Repos that are, have a lot of history, have a lot of people, but they don't really represent like what it is to do day-to-day work for us that are building like apps and client thi…”
Nov 28, 2024 neutral
Nov 28, 2024
Dec 13, 2024 negative
Ramachandran: SWE-bench and HumanEval do not reflect real professional engineering
“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have sweet bench. That's cool. No actual Professional work looks like Sweebench. Like, human eval, same thi…”
Dec 21, 2024 bullish
Dec 25, 2024 bearish
Neubig: SWE-bench Scores Are Inflated By Training Data Contamination
“Sweebench is on popular open source repos and all of these popular open source repos were included in the training data for all of the language models. And so, the language models already know these repos. In some cases, the language models already know the in…”
Jan 1, 2025 positive
Jan 24, 2025 positive
Jan 28, 2025
Automating AI trace inspection is impossible; manual review remains mandatory
“And really you cannot avoid this last part. Like I've tried a lot to automate parts of this by I've tried a lot to automate it to remove the need for me to actually like manually inspect all of these traces. And I'm here to tell you like today, that is still i…”
Jan 28, 2025
Topping SWE-bench requires multi-trajectory sampling and high compute costs
“You know, if you want to get to the top of the leaderboard at this point, you probably need a strategy like that somewhere and be willing to, like, pay the cost, whatever it is, but it's also, like, it's really important, like, to be able to move up six percen…”
Jan 28, 2025 neutral
Jan 28, 2025
Debug agent regressions by qualitatively clustering failures across execution traces
“So it's really, like, I'll flip through these traces and kind of, like for each one, I'll write down notes about, like, what I thought went wrong there, and I'll do that for, like, say, 20 or so, and then I kind of go, okay, what's the biggest problem that we …”
Jan 28, 2025 bullish
Jan 28, 2025
Mar 7, 2025 neutral
A 90% SWE-Bench Score Can Still Fall Flat in Customer Environments
“Autonomous coding benchmarks, let's say, like Sweetbench, are useful. I'm not going to discount them. They are useful. But let's say, you know, 90% on Sweetbench could still mean something that just falls over flat within a customer setting.”
Apr 2, 2025 positive
Apr 2, 2025 neutral
Apr 2, 2025
Apr 2, 2025 bearish
Gur-Ari: SWE-bench fails to effectively test true codebase understanding
“What we found with SuiteBench is an interesting benchmark, very useful for doing some things like refining prompts, for example, and trying out different tools. Specifically for codebase understanding, in the beginning, we were hopeful That it will tell us mor…”
May 16, 2025 negative
Embiricos: Benchmark-Passing SWE Agent Outputs Are Often Unmergeable in Practice
“Because if you look at a lot of, like, Sweebench passing, like, outputs from, like, an agent, they're not really, like, PRs that you would merge, because, like, the code style might be, like, different. Like, it works, but the code style is different.”
Jun 25, 2025 bullish
Zach Lloyd: Warp's coding agent will likely top the TBench benchmark
“Basically, state of the art on SweetBench, I think we will, again, I don't want to be quoted here, we can maybe edit this later, but like, we'll probably be number one or close to it on TBench also, which is the terminal benchmark, which really we should be th…”
Jul 28, 2025 negative
Ramachandran: SWE-bench and HumanEval do not reflect real professional software engineering
“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have SweeBench, that's cool, no actual Professional work looks like Sweebench, like human eval, same thing.”
Jul 28, 2025 negative
Mohan: Squeezing the last 10% from AI benchmarks is counterproductive
“Once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is, like, counterintuitive to the actual goal of what the benchmark was. Like, you probably should find a new hill to climb, rather than sort of p-hacking or really op…”
Sep 30, 2025 positive
Dec 31, 2025 positive
Yang: TerminalBench enables more creative eval design than SWE-bench
“As we bench, you're confined in some sense to the domain of issues and PRs that already exist which I think has its benefits of being close to reality and natural, but I think with Terminal Bench, there's a lot of creativity that you can infuse into that”
Dec 31, 2025 negative
Dec 31, 2025 neutral
Dec 31, 2025 positive
Dec 31, 2025 positive
Yang: SWE-bench adoption took off only after Cognition's Devin launch
“You know, we put it out October, 22, 23, and then people didn't really touch it too much. And then, of course, like, Cognition came on the scene, and Devon was an amazing release, and I think after that, it kind of kicked off the arms.”
Dec 31, 2025 negative
Feb 23, 2026 negative
Watkins: Over half of SWE-bench problems investigated by OpenAI had test flaws
“In over half of the problems that were investigated in that deep dive, there was one problem or the other. I think the most common problem are, like, overly narrow tests where there's some particular implementation detail that the tests were looking for but wa…”
Feb 27, 2026 bearish
Becker: AI Coding Fails at Merge Readiness Despite High SWE-bench Scores
“Maybe one that I'll call out there is this difference between whether models pass unit tests, whether they succeed by, you know, SWE bench-like scoring kind of meter-like scoring, benchmark-style scoring, versus whether their solution would be merged into main…”