Gur-Ari uses human contractors because chat and agent evaluations resist automation
Guy Gur-Ari · The #1 SWE-Bench Verified Agent · Apr 2, 2025 · at 7:33
Guy Gur-Ari of Augment Code discusses the necessity of human contractor evaluations for complex AI agent behavior on the Latent Space podcast.
0:00 / 0:11exact quote · 11.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The last thing I can mention is we use contractors for evaluation where we cannot Do automatic evaluation. So with chats and agents, it becomes way harder to do things automatically. And so we use contractors for that.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Guy Gur-Ari
Gur-Ari: SWE-bench fails to effectively test true codebase understanding
“What we found with SuiteBench is an interesting benchmark, very useful for doing some things like refining prompts, for example, and trying out different tools. Specifically for codebase understanding, in the beginning, we were hopeful That it will tell us mor…”
Sequential Thinking MCP outperformed Claude 3.7 native reasoning mode in evaluations
“We tried reasoning mode as well with the new three seven. And we didn't see that much of a bump in performance. We don't know if this is something that's code specific or not. I don't have an insight. We tried both and yeah, sequential thinking worked better.”
Developers will eventually spend 80% of their time controlling agents outside IDEs
“I think in the future, at some point, my guess is that the IDE is going to become less of the focal point and more like an app that you can launch when you need to dig in deeper, but you spend most of your time away from it. So maybe 80% of your time is in a w…”
Gur-Ari: Augment Code Achieved #1 on SWE-Bench Verified
“We just made number one on Sweepbench. So for us Sweepbench has been a useful tool for exploring how can we get the most out of agents. And so we were able to get the best result on Sweepbench verified right now.”
Augment's #1 SWE-Bench agent uses off-the-shelf models, unlike their custom product
“The generation models for Sweebench, it's all off the shelf models. For the product, it includes our own custom trained models that help the model with, that help the agent with code-based understanding.”
Ground-truth evaluations without code execution provide significant mileage for AI
“So code execution, like checking the correctness of solutions and running tests automatically can help, although not, although you can get a lot of mileage out of evals that don't have code execution in them that just compare against ground truth.”