Swyx: OpenHands is ranked number one on SWE-bench Full
“He started open hand is currently still number one on SweetBench full, which is the hardest one.”
Swyx: SWE-bench resolution rates surged from 13% to ~50% in 2024
“Keep in mind, we started the year at 13%. And so now we're about 50 open hands is around there.”
Neubig: Browsers, Terminals, And Code Editors Constitute The Core Agent Toolset
“Let's say I gave you a web browser and a terminal or a file system and the ability to edit text or code. What could you do with that? Everything. Yeah, probably a lot of things. This is like 99% of my, you know, daily daily life, I guess when I'm working. So I…”
Neubig: Arbitrary Python Execution Beats Individual Agent Tool Calls
“And the method that we adopt in open hands instead is we provide these tools, but we provide them by just giving a coding agent the ability to call arbitrary Python code. And in the arbitrary Python code, it can call these tools. We expose these tools as APIs …”
Neubig: OpenHands Equips Its Agent With Only Five Or Six Tools
“We're kind of extreme. And we're only giving the agent five tools or maybe six tools.”
Neubig: Nobody Has A Good Answer For Human-Agent Interface Design
“I don't think anybody has a good answer to this, and I don't think we have a good answer to this”
Neubig: OpenHands Uses A Single Prompt With No Multi-Agent Systems
“So in open hands, we do very light planning. We have a single prompt. We don't have any multi agent systems.”
Neubig: All Hands AI Is Releasing A Unified Agent Benchmark
“We don't have benchmarks that test whether agents can code and do web navigation. But we're working on that and hoping to release something in the next week or two.”
Neubig: Anthropic's MCP Duplicates Existing APIs With Little Added Value
“We already have an API for GitHub. So why do we need an MCP for GitHub, right? You know, like GitHub has an API. The GitHub API is evolving. We can look up the GitHub API documentation. So it seems like kind of duplicated a little bit. And also they have a set…”
Neubig: Inadequate Information Gathering Is The Biggest Agent Failure Mode
“So I think actually probably the biggest thing that it fails at is. Or that our agent plus Claude fails at is insufficient information gathering before trying to solve the task, and so if you provide all, if you provide instructions that it should do informati…”