Nov 28, 2024 · 1h 11m · latent-space
The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Anthropic's Erik Schluntz discusses the engineering principles behind Claude 3.5 Sonnet's state-of-the-art SWE-bench performance, mistake-proof tool design, the launch of Computer Use, and pragmatic lessons bridged from physical robotics.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 8.4% of the talking time here. How this is scored →
The hosts as informed peer 6.3 Guest teaching 5.2 Guest disagreement 1.7 The hosts pushing back 2.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 42:54 Framework skepticism and critique of agent libraries
Erik takes a sharp, contrarian stance against popular agent frameworks, arguing they encourage overly complex multi-agent architectures that obfuscate raw model outputs and hinder debugging.
Hardest push from the hosts ▶ 1:08:24 Swyx challenges skepticism around autonomous vehicle business modelsSwyx directly refuses Erik's gloomy framing of self-driving economics by citing Waymo's public metric of exceeding 100k trips per week and an estimated $130M revenue run rate.
Biggest teaching moment ▶ 37:51 Poka-yoke and mistake-proofing agent toolsErik explains the concept of poka-yoke in agent tool design, illustrating how enforcing absolute file paths and non-blocking command execution prevents model failure loops.
The host holds their own ▶ 42:00 Swyx traces SWE-agent and ReAct academic lineageSwyx demonstrates deep insider familiarity with the research ecosystem by connecting SWE-agent, Princeton, and Shunyu Yao's ReAct framework directly to Anthropic's reference implementation.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Erik Schluntz's Journey from Robotics to Anthropic | 4 | 3 | 1 | 0 | A friendly introductory exchange covering Erik's career pivot from Cobalt Robotics and SpaceX to Anthropic. The hosts prompt Erik for biographical context without any friction or debate. | |
| SWE-bench Overview and Real-World Software Engineering Evaluation | 6 | 5 | 1 | 2 | Swyx accurately deconstructs SWE-bench's scope to clarify it covers a narrow subset of Python repositories rather than general software engineering. Erik explains why repo-level benchmarks represent real development far better than isolated interview-style HumanEval puzzles. | |
| SWE-bench Verified, Failure Modes, and Prompt Tradeoffs | 7 | 6 | 2 | 3 | Erik details key failure modes in SWE-bench Verified, explaining how complex prompting hurt overall benchmark performance on easy tasks. Swyx contributes detailed insights into meta-prompting and references Dario Amodei's thoughts on model conciseness. | |
| Minimal Agent Architecture, Self-Correction, and Agentic Search | 7 | 5 | 1 | 2 | Alessio and Swyx press Erik on architectural choices such as context pruning, agentic search, and vector databases. Erik explains Anthropic's philosophy of running minimal agent loops and relying on model self-correction. | |
| Designing Agent-Computer Interfaces and Mistake-Proofing Tools | 6 | 7 | 2 | 1 | Erik delivers a masterclass on designing agent-computer interfaces (ACIs), demonstrating why string replacement outperforms diff generation. The hosts engage with sandbox implementations and environment constraints. | |
| Interactive Agent Benchmarking, Planning Phases, and XML Tags | 7 | 4 | 1 | 2 | The hosts and Erik discuss the advantages of interactive planning stages (citing Devin and Replit) and benchmark limitations. Swyx and Erik discuss the structural efficiencies of XML tags over JSON escaping for code generation. | |
| Tool Iteration, SWE-agent Foundation, and Framework Skepticism | 7 | 6 | 4 | 2 | Erik strongly advises against using popular agent frameworks, arguing they add unnecessary complexity and hide raw prompts. Swyx showcases domain knowledge by mapping the Princeton lineage behind SWE-agent and the ReAct paradigm. | |
| Context Scaling, Multi-Model Orchestration, and Coding Startups | 6 | 4 | 1 | 2 | The discussion covers multi-model orchestration between Haiku and Sonnet to save context tokens, alongside startup benchmark dynamics. Erik notes that startups often avoid public SWE-bench submissions to protect proprietary execution traces. | |
| Claude Computer Use and Frictionless Enterprise Tool Integration | 6 | 6 | 1 | 2 | Erik presents an analogy from Cobalt Robotics comparing elevator button-pushing arms to GUI computer use as zero-friction integration. Swyx compares this to RPA script generation and traditional enterprise remote workflows. | |
| AI Robotics Realities: Diffusion Policies, Reliability, and Hardware | 7 | 6 | 3 | 4 | Erik outlines severe practical and economic bottlenecks in robotics and self-driving cars, highlighting hardware variance and 99.9% reliability thresholds. Swyx challenges Erik's skepticism on Waymo by citing their ride volume and revenue figures. |
Statements from this episode (24)
Anthropic releases exact tools and prompt used for SWE-bench agent
“With this blog post we released on SweetBench, we released the exact tools and the prompt that we gave the model to be able to do well.”
Schluntz: SWE-bench reflects real engineering by requiring repository navigation
“Sweebench, you're starting in the context of an entire repository. And so it adds this entirely new dimension to the problem of finding the relevant files. And, you know, this is a huge part of real engineering”
Schluntz: Traditional coding evals remain useful alongside SWE-bench
“I think there's definitely a space for these more traditional coding evals that are sort of easy to implement, quick to run and do get you some signal. And maybe hopefully there's just sort of harder versions of human eval that get created.”
Schluntz: SWE-bench Verified was created in partnership with OpenAI
“SweetBench Verified was actually made in partnership with OpenAI, and they hired humans to go review all these tasks and pick out a subset to try to remove any obstacle like this that would make the tasks impossible.”
Schluntz: Language models prefer small diffs over major refactors
“Language models frequently will produce like a smaller diff when possible, rather than trying to do a big refactor.”
Schluntz: Smarter AI models require less agent scaffolding
“And I think like the smarter the models are, the less you need that kind of extra scaffolding.”
Anthropic: Tool engineering mattered more than prompt engineering for SWE-bench
“I would say actually we did more engineering of the tools than the overall prompt.”
Schluntz: String replacement is the most reliable file-editing tool for LLMs
“We did a few different experiments with like different ways to specify how to edit a file and string replace. Basically the model has to write out the existing version of the string and then a new version, and that just gets swapped in. We found that to be the…”
Schluntz: Full file regeneration is most accurate for LLMs but cost-prohibitive
“Having the model fully regenerate files. That one is actually the most accurate, but it takes so many tokens. And if you're in a very big file, it's cost prohibitive.”
Schluntz: Developers must iterate on agent tools rather than just prompts
“You need to iterate on tools rather than just a prompt. And I think a lot of people, when they make tools for an LLM, they kind of treat it like they're just writing an API for a computer and it's sort of very minimal. It's sort of just the bare bones of what …”
Schluntz: Agent-Computer Interfaces require as much design effort as frontend UI
“Think about how much human effort goes into designing human interfaces for things. Like it's incredible. This is like entirely what front end is about. Is creating better interfaces to kind of do the same things. And I think that same amount of attention and e…”
Schluntz: Real-World Coding Agent Workflows Will Be Interactive, Not One-Shot
“So I think that like real tasks are going to be much more interactive with the agent rather than this kind of like one shot system.”
Schluntz: Initial Planning Steps Improve Agent Performance and Human Trust
“I think for agents in general, like having a planning step at the beginning, one, just having that plan will improve performance on the downstream task, just because it's kind of like a bigger chain of thought, but also it's just such a better UX. It's way eas…”
Schluntz: JSON Escaping Overhead Degrades LLM Performance Across the Board
“Like if you're trying to output a code in JSON, there's a lot of extra escaping that needs to be done. And that actually hurts model performance across the board. Where versus like if you're in just a single XML tag, there's none of that sort of escaping that …”
Anthropic's Schluntz: Avoid agent frameworks and start from scratch with raw prompts
“I think with agent frameworks in general, they can certainly save you some like boilerplate, but I think there's actually this like downside of making agents too easy, where you end up very quickly, like building a much more complex system than you need. And s…”
Schluntz: Production AI agent applications will be bespoke, not off-the-shelf
“You know, I think that might be useful for hobbyists and demos, but the ultimate end applications are going to be bespoke. And so we just want to make sure that the model's great at any tool that it uses”
Schluntz: Anthropic will not focus on further SWE-bench submissions
“You know, we're not going to go and do lots more submissions to sweet bench and try to try to prompt engineer this and build a bigger system. We want people to like the ecosystem to do that on top of our models.”
Wang: Cosine scored 50+ on SWE-bench Full but was rejected over traces
“So we had Cosign on they had a, like a 50 something on full, on Sweebench full, which is the hardest one. And they were rejected because they didn't want to submit their traces.”
Schluntz: Computer use is primarily a low-friction way to implement tool use
“I think most broadly, not just for like new things that weren't possible before, but as a much lower friction way to implement tool use.”
Swyx: APIs serve high-volume tasks while computer use handles the long tail
“I think there could be a separation between maybe like the high volume use cases you want APIs and then the long tail you want computer use.”
Schluntz: AI robotics today is where autonomous driving was 10 years ago
“I think where we are right now is where self-driving cars were 10 years ago. I think we have very cool demos that work. I mean, 10 years ago, you had videos of people driving a car on the highway, driving a car, you know, on a street with a safety driver, but …”
Schluntz: Reliability, not demo capability, is the bottleneck for robotics
“I think that for robotics, the limiting factor is going to be reliability. That these models are really good at doing these demos of, like, doing laundry or doing dishes. If they only work 99% of the time, like, that sounds good, but that's actually really ann…”
Schluntz: High vehicle costs make Waymo's per-car profitability doubtful
“Those cars are expensive. It's not about if you can hit profitability, it's about your cash conversion cycles. Like is building one Waymo, like how cheap can you make that compared to like how much you're earning sort of as the equivalent of what an Uber drive…”
Schluntz: Trust and Auditability Will Be LLM Agents' Biggest Bottleneck
“The biggest limiting thing will start to become like, do people trust the output of these agents? And like, how do you trust the output of an agent that did five hours of work for you and is coming back with something? And if you can't find some way to trust t…”