Gur-Ari: Augment Code Achieved #1 on SWE-Bench Verified
Guy Gur-Ari · The #1 SWE-Bench Verified Agent · Apr 2, 2025 · at 1:24
Guy Gur-Ari, founding member at Augment Code, explains Augment's autonomous agent benchmark rankings on the Latent Space podcast.
0:00 / 0:13exact quote · 13.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We just made number one on Sweepbench. So for us Sweepbench has been a useful tool for exploring how can we get the most out of agents. And so we were able to get the best result on Sweepbench verified right now.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Guy Gur-Ari
Gur-Ari: SWE-bench fails to effectively test true codebase understanding
“What we found with SuiteBench is an interesting benchmark, very useful for doing some things like refining prompts, for example, and trying out different tools. Specifically for codebase understanding, in the beginning, we were hopeful That it will tell us mor…”
Sequential Thinking MCP outperformed Claude 3.7 native reasoning mode in evaluations
“We tried reasoning mode as well with the new three seven. And we didn't see that much of a bump in performance. We don't know if this is something that's code specific or not. I don't have an insight. We tried both and yeah, sequential thinking worked better.”
Developers will eventually spend 80% of their time controlling agents outside IDEs
“I think in the future, at some point, my guess is that the IDE is going to become less of the focal point and more like an app that you can launch when you need to dig in deeper, but you spend most of your time away from it. So maybe 80% of your time is in a w…”
Augment's #1 SWE-Bench agent uses off-the-shelf models, unlike their custom product
“The generation models for Sweebench, it's all off the shelf models. For the product, it includes our own custom trained models that help the model with, that help the agent with code-based understanding.”
Ground-truth evaluations without code execution provide significant mileage for AI
“So code execution, like checking the correctness of solutions and running tests automatically can help, although not, although you can get a lot of mileage out of evals that don't have code execution in them that just compare against ground truth.”