Insight
Anthropic's Schluntz: Avoid agent frameworks and start from scratch with raw prompts
“I think with agent frameworks in general, they can certainly save you some like boilerplate, but I think there's actually this like downside of making agents too easy, where you end up very quickly, like building a much more complex system than you need. And s…”
Insight
Schluntz: Smarter AI models require less agent scaffolding
“And I think like the smarter the models are, the less you need that kind of extra scaffolding.”
Insight
Schluntz: JSON Escaping Overhead Degrades LLM Performance Across the Board
“Like if you're trying to output a code in JSON, there's a lot of extra escaping that needs to be done. And that actually hurts model performance across the board. Where versus like if you're in just a single XML tag, there's none of that sort of escaping that …”
Opinion
Schluntz: AI robotics today is where autonomous driving was 10 years ago
“I think where we are right now is where self-driving cars were 10 years ago. I think we have very cool demos that work. I mean, 10 years ago, you had videos of people driving a car on the highway, driving a car, you know, on a street with a safety driver, but …”
Opinion
Schluntz: High vehicle costs make Waymo's per-car profitability doubtful
“Those cars are expensive. It's not about if you can hit profitability, it's about your cash conversion cycles. Like is building one Waymo, like how cheap can you make that compared to like how much you're earning sort of as the equivalent of what an Uber drive…”
Disclosure
Anthropic: Tool engineering mattered more than prompt engineering for SWE-bench
“I would say actually we did more engineering of the tools than the overall prompt.”
Assertion Not checkable as stated
Schluntz: String replacement is the most reliable file-editing tool for LLMs
“We did a few different experiments with like different ways to specify how to edit a file and string replace. Basically the model has to write out the existing version of the string and then a new version, and that just gets swapped in. We found that to be the…”
Insight
Schluntz: Developers must iterate on agent tools rather than just prompts
“You need to iterate on tools rather than just a prompt. And I think a lot of people, when they make tools for an LLM, they kind of treat it like they're just writing an API for a computer and it's sort of very minimal. It's sort of just the bare bones of what …”
Opinion
Schluntz: Agent-Computer Interfaces require as much design effort as frontend UI
“Think about how much human effort goes into designing human interfaces for things. Like it's incredible. This is like entirely what front end is about. Is creating better interfaces to kind of do the same things. And I think that same amount of attention and e…”
Insight
Schluntz: Initial Planning Steps Improve Agent Performance and Human Trust
“I think for agents in general, like having a planning step at the beginning, one, just having that plan will improve performance on the downstream task, just because it's kind of like a bigger chain of thought, but also it's just such a better UX. It's way eas…”
Prediction Not checkable as stated
Schluntz: Production AI agent applications will be bespoke, not off-the-shelf
“You know, I think that might be useful for hobbyists and demos, but the ultimate end applications are going to be bespoke. And so we just want to make sure that the model's great at any tool that it uses”
Insight
Schluntz: Reliability, not demo capability, is the bottleneck for robotics
“I think that for robotics, the limiting factor is going to be reliability. That these models are really good at doing these demos of, like, doing laundry or doing dishes. If they only work 99% of the time, like, that sounds good, but that's actually really ann…”
Insight
Schluntz: SWE-bench reflects real engineering by requiring repository navigation
“Sweebench, you're starting in the context of an entire repository. And so it adds this entirely new dimension to the problem of finding the relevant files. And, you know, this is a huge part of real engineering”
Opinion
Schluntz: Traditional coding evals remain useful alongside SWE-bench
“I think there's definitely a space for these more traditional coding evals that are sort of easy to implement, quick to run and do get you some signal. And maybe hopefully there's just sort of harder versions of human eval that get created.”
Insight
Schluntz: Language models prefer small diffs over major refactors
“Language models frequently will produce like a smaller diff when possible, rather than trying to do a big refactor.”
Insight
Schluntz: Full file regeneration is most accurate for LLMs but cost-prohibitive
“Having the model fully regenerate files. That one is actually the most accurate, but it takes so many tokens. And if you're in a very big file, it's cost prohibitive.”
Prediction Not checkable as stated
Schluntz: Real-World Coding Agent Workflows Will Be Interactive, Not One-Shot
“So I think that like real tasks are going to be much more interactive with the agent rather than this kind of like one shot system.”
Insight
Schluntz: Computer use is primarily a low-friction way to implement tool use
“I think most broadly, not just for like new things that weren't possible before, but as a much lower friction way to implement tool use.”
Prediction Not checkable as stated
Schluntz: Trust and Auditability Will Be LLM Agents' Biggest Bottleneck
“The biggest limiting thing will start to become like, do people trust the output of these agents? And like, how do you trust the output of an agent that did five hours of work for you and is coming back with something? And if you can't find some way to trust t…”
Disclosure
Anthropic releases exact tools and prompt used for SWE-bench agent
“With this blog post we released on SweetBench, we released the exact tools and the prompt that we gave the model to be able to do well.”
Assertion Supported
Schluntz: SWE-bench Verified was created in partnership with OpenAI
“SweetBench Verified was actually made in partnership with OpenAI, and they hired humans to go review all these tasks and pick out a subset to try to remove any obstacle like this that would make the tasks impossible.”
Disclosure
Schluntz: Anthropic will not focus on further SWE-bench submissions
“You know, we're not going to go and do lots more submissions to sweet bench and try to try to prompt engineer this and build a bigger system. We want people to like the ecosystem to do that on top of our models.”
