SWE Bench

“At the time the best known was like Three or four percent or something like that, but obviously, yeah, 13% still means you fail, you know, 87% of the time.”

“And we built you know, soda coding agents, which became world number one on SweetBench you know, in two months of time.”

“Maybe one that I'll call out there is this difference between whether models pass unit tests, whether they succeed by, you know, SWE bench-like scoring kind of meter-like scoring, benchmark-style scoring, versus whether their solution would be merged into main…”

“In over half of the problems that were investigated in that deep dive, there was one problem or the other. I think the most common problem are, like, overly narrow tests where there's some particular implementation detail that the tests were looking for but wa…”

“You know, we put it out October, 22, 23, and then people didn't really touch it too much. And then, of course, like, Cognition came on the scene, and Devon was an amazing release, and I think after that, it kind of kicked off the arms.”

“I got an email about, like, two weeks ago. I think it was from, I think it was from Walden. It was like, hey, you know, we have a good number on it.”

“I don't like unit tests as a form of verification, and I also think there's an issue with Sweepbench where all of the task instances are independent of each other.”

“Like, you know, you can do well on those first, and then sort of graduate to the multi-turn expensive stuff.”

“I think they're all, the models are all kind of attempting and saying, like, oh, I did it, you know, so maybe not great.”

“As we bench, you're confined in some sense to the domain of issues and PRs that already exist which I think has its benefits of being close to reality and natural, but I think with Terminal Bench, there's a lot of creativity that you can infuse into that”

“Sweebench is the main benchmark used to test whether AI is good at software engineering tasks, and we're almost saturating that. So last year we're at like maybe five percent early 24 or less, and now we're like 82% or something like that with cloths on at 4.5…”

“So we're, you know, we're number one on one of the coding benchmarks, which is terminal bench. We're number three now on sweet bench. So the two public coding benchmarks.”

“We moved recently from roughly 72 to roughly 78 in Sweepbench”

“As recently as a year ago, I think we were under 20% or something like that as a field.”

“Even when it was already outperforming Opus, for example, on sweet bench, people still didn't feel it was better, but then it continued to train and it was like now better than Opus and people don't want to switch back.”

“I don't know, I think we were at, like, 10% last year, and now we're at, like, 70% and 80%.”

“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have SweeBench, that's cool, no actual Professional work looks like Sweebench, like human eval, same thing.”

“Once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is, like, counterintuitive to the actual goal of what the benchmark was. Like, you probably should find a new hill to climb, rather than sort of p-hacking or really op…”

“Claude IV Opus gets a 72.5 on that. Kimi K-II gets 65.8, so not far behind. And just to, you know, give some context, Deep Seek V-III, which everybody was going crazy over, gets a 38.”

“Basically, state of the art on SweetBench, I think we will, again, I don't want to be quoted here, we can maybe edit this later, but like, we'll probably be number one or close to it on TBench also, which is the terminal benchmark, which really we should be th…”

“Because if you look at a lot of, like, Sweebench passing, like, outputs from, like, an agent, they're not really, like, PRs that you would merge, because, like, the code style might be, like, different. Like, it works, but the code style is different.”

“The generation models for Sweebench, it's all off the shelf models. For the product, it includes our own custom trained models that help the model with, that help the agent with code-based understanding.”

“What we found with SuiteBench is an interesting benchmark, very useful for doing some things like refining prompts, for example, and trying out different tools. Specifically for codebase understanding, in the beginning, we were hopeful That it will tell us mor…”

“Ensembling is usually not the kind of thing that just solves a benchmark, just also in other domains, you typically can get like, yeah, and this, and first we bench probably a few percentage points is my guess.”

“We actually open sourced our implementation of Sweebench. So if you're curious how we got to number one, you'll be able to go see all the details of how we did it.”

“As the models get better in reasoning we're going to get closer to a hundred percent of this VBench, which is that benchmark out of 12 repos open source Python repos a team in Princeton identified 2200 or so issue pull request pairs. Effectively, all the model…”

“Autonomous coding benchmarks, let's say, like Sweetbench, are useful. I'm not going to discount them. They are useful. But let's say, you know, 90% on Sweetbench could still mean something that just falls over flat within a customer setting.”

“There's a, there's this benchmark called SWE bench. So software engineering bench. And that's what everyone's competing on. We don't really compete on that. And the reason is because we are building for a different audience.”

“It solves, like, something like 57% of problems with a single Rollout and then using parallel rollouts and selecting the best one. With other techniques, we get something like 64%.”

“You can see in the course of this, I did something like a thousand evals.”

“And really you cannot avoid this last part. Like I've tried a lot to automate parts of this by I've tried a lot to automate it to remove the need for me to actually like manually inspect all of these traces. And I'm here to tell you like today, that is still i…”

“So it's really, like, I'll flip through these traces and kind of, like for each one, I'll write down notes about, like, what I thought went wrong there, and I'll do that for, like, say, 20 or so, and then I kind of go, okay, what's the biggest problem that we …”

“The Google submission down below, I think ran thousands of trajectories and then has a strategy for choosing the best.”

“You know, if you want to get to the top of the leaderboard at this point, you probably need a strategy like that somewhere and be willing to, like, pay the cost, whatever it is, but it's also, like, it's really important, like, to be able to move up six percen…”

“And then we also on this podcast, we interviewed Cosign that actually fine tune four O to, on three bench, sorry to achieve data on three bench. And that score was actually higher than O one when it came out.”

“Keep in mind, we started the year at 13%. And so now we're about 50 open hands is around there.”

“Sweebench is on popular open source repos and all of these popular open source repos were included in the training data for all of the language models. And so, the language models already know these repos. In some cases, the language models already know the in…”

“If you recall, like a year ago, the point of view on sweet bench was like, it was impossible to surpass. Team percent or so. And I think the whole industry now considers that if not trivial accessible.”

“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have sweet bench. That's cool. No actual Professional work looks like Sweebench. Like, human eval, same thi…”

“I would say actually we did more engineering of the tools than the overall prompt.”

“You know, we're not going to go and do lots more submissions to sweet bench and try to try to prompt engineer this and build a bigger system. We want people to like the ecosystem to do that on top of our models.”

“And I'll say the critique here is that these are, like, really robust you know, Repos that are, have a lot of history, have a lot of people, but they don't really represent like what it is to do day-to-day work for us that are building like apps and client thi…”

“If we just use GPT-IV plus RAG, what do we get? It's, like, a measly three percent. And then up to the most recent submissions where they get up to 43%.”

“The diff between that and something like Devin is all in like, sort of like the agent scaffold or the agent code, right? So that's, what's really exciting about this stuff. It shows off what you can do just from prompting and just from adding tools.”

“Everyone is using the same base models here, or the same model APIs period. I don't know if anyone except for cosine was even training, but like the diff here is all in prompting.”

“The entire test split here is public. So you can do things like just overfit to the patches in the test set. You can do things like, let me add at runtime, pull the PR and just get the answer and just use it.”

“They actually require you to submit these things called trajectories that are proving what you did and that you're not cheating. And a lot of this ends in controversy. A lot of this ends in people say, well, we don't want to submit at all because if we review …”

“Sometimes it feels like you try to read the issue. And you're just like, okay, even if I was like an Oracle or like some sort of God coder, I couldn't, I would not be able to solve this because there's some context that was left out here. There's something tha…”

“And then there's another one called Honeycomb, which also scored really well, and I think is number one on the full set.”

“There's something nuanced about this data set in particular where all the edits are super short and it's like a prior that you can put into your code. But if you don't have that, then yeah, like agents tend to just keep making obnoxiously long edits”

“So it was obviously great to see, like, we still are better than O-one out of the box. You know, even with an older model, and I'm sure that that, that Delta will continue to grow once we're able to train O-one and once we've done more work on our dataset usin…”

“I know Sweebench is, like, the most commonly talked about thing, and honestly, it's a very, it's an amazing project, but one of the things we've learned the most from actually shipping this product to users is, it's a pretty bad proxy at telling us how compete…”

“And I think part of the reason that Sweetbench is so popular now is it kind of hits the balance between these three dimensions, right? Easy to evaluate and being actually practical and being scalable.”

“It was actually a very useful tool in building Genie because beforehand it was like, yes, vibe check this thing and see if it's useful. And then all of a sudden you have an actual measure to see like, couldn't it do software engineering? Not the best measure, …”

“What the, very crudely, what the pre-trained models are reading is they're reading those final diffs and they're Emulating that and then being able to output it. Right. But of course it's a super lossy thing, a PR. You have no idea why or how, for the most par…”

“The way that we Decided we want to try to extract what happened in the past, like as forensically as possible, has been and is currently like one of the main things that we focus all our time on. Because doing that as we're getting as much signal out as possib…”

“I think it's fairly obvious that with the foundational models, like you can ask them to think by step by step and ask them to plan and stuff, but that isn't enough because if you look at how those models score on these benchmarks and then they're not They're n…”

“Performance of Jeannie over the length of the context window degrades fairly linearly. So actually, I actually broke it down by probability of solving a SWE bench issue. Given the number of tokens of the context window at 60 K, it's basically .5. So if you go …”

“For the moment, as a closed source company, like fighting for an edge, we've decided not to publish that information for that exact reason. I don't want someone basically taking my tragedies and then taking a model that's suing them in GA and just distilling i…”

“The patches are still public. So like the, dare I say, traditional Sweebench submission, you can go to a GitHub repo and see it and run them for yourself and verify that the numbers come out correctly.”