Ball: The Sourcegraph Amp team does not use formal evals
Thorsten Ball · Amp: The Emperor Has No Clothes · Sep 25, 2025 · at 1:18:11
Thorsten Ball, Amp lead at Sourcegraph, discusses how the Amp engineering team evaluates product quality without relying on formal automated benchmark evals.
0:00 / 0:15exact quote · 15.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think we don't have any set evals. We don't. And this was controversial up until a week ago, I think, when I think Boris from Or two weeks ago from Anthropix that they don't have evals for the coding agent too. But we don't, and we haven't had them.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Thorsten Ball
Ball: Amp core team has 8 people, ships 15 times daily without code reviews
“I think we're around eight people now on the AMP core team, and we still don't do formal code reviews. We still push to main. We still ship 15 times every day.”
Ball: Complex sub-agent workflows will result in user hangovers
“A lot of the features what we see, you know, where people build like elaborate workflows, like I have my custom slash commands and they trigger custom Custom sub-agents and they in turn trigger custom MCP tool calls behind which again another model is doing in…”
Ball: AI coding tools must optimize harness and scaffolding per model
“And people are now also waking up to the fact that it's not just the model, it's the system prompt, it's the tools, it's the harness. The scaffolding around the model. So I can give you the choice to use Gemini 2.5 in, in AMP, but without the system prompting …”
Ball: Foundation models will become background implementation details in AI tools
“So I think we're going towards a future where the model will become an implementation detail to some sense, and we will end up on a different abstraction layer.”
Ball: Coding agents should outsource typing, not engineering thinking
“The failure mode of outsourcing the thinking, but not the typing, which I think it should be the opposite. You still have to know engineering. You still have to know how to program. You still have to know your application and its architecture, how it's deploye…”