Dec 25, 2024 · 51m · latent-space
Best of 2024 in Agents (from #1 on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Professor Graham Neubig shares technical insights and architectural lessons from building OpenHands, an open-source autonomous coding agent framework. Through live demonstrations and research findings, he examines agent-computer interfaces, benchmarking challenges, memory systems, and the future of open-source AI development.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 0.9 Guest teaching 1.9 Guest disagreement 0.4 The hosts pushing back 0.1
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 41:05 Neubig questions the necessity of Anthropic's MCP
Neubig pushes back against the industry hype surrounding Anthropic's Model Context Protocol, arguing that creating bespoke agent protocols duplicates existing developer APIs like GitHub's.
Hardest push from the hosts ▶ 46:41 Audience member presses Neubig on skill libraries vs playbooksAn audience member challenges Neubig's framing on memory retrieval by drawing explicit parallels to Voyager's skill library and Devin's playbook system.
Biggest teaching moment ▶ 32:41 Neubig breaks down SWE-bench data leakageNeubig explains the root cause behind benchmark score inflation, demonstrating that prominent open source repositories and specific GitHub issues leaked directly into foundation model training corpora.
The host holds their own ▶ 46:41 Audience member demonstrates domain expertise on agent memory paradigmsThe audience questioner cites Voyager's code-based skill persistence and Devin's architecture to probe the limits of language-to-code RAG.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Live Demos of Real-World Coding Agent Tasks | 0 | 0 | 0 | 0 | Graham Neubig delivers a solo presentation and live demo showcasing OpenHands solving data science, API integration, and repository monitoring tasks. There is no host interaction during this demo segment. | |
| Agent-Computer Interfaces and Code Execution Paradigms | 0 | 0 | 0 | 0 | Neubig explains the agent-computer interface architecture, arguing why executing arbitrary Python scripts in Jupyter/bash outperforms granular API tool calls. The presentation is entirely uninterrupted. | |
| Designing Human-Agent Interfaces Across Developer Environments | 0 | 0 | 0 | 0 | Neubig discusses user interface paradigms for coding agents, including chat interfaces, GitHub issue bots, and headless runtime execution. | |
| Selecting Foundational Models for Agentic Performance | 0 | 0 | 0 | 0 | Neubig assesses foundation model capabilities, highlighting Claude's superior error recovery and self-correction compared to GPT-4o. | |
| Planning Strategies and Single-Agent Architecture | 0 | 0 | 0 | 0 | Neubig presents his case for single-agent systems over rigid multi-agent pipelines, citing greater flexibility when execution plans deviate. | |
| Workflow Memory, Pre-Exploration, and Search Trajectories | 0 | 0 | 0 | 0 | Neubig details recent academic advances in workflow memory, environment exploration such as Bagel and Agentless, and tree search strategies for coding agents. | |
| Evaluation Benchmarks from Sanity Checks to Realistic Environments | 0 | 0 | 0 | 0 | Neubig outlines the landscape of evaluation benchmarks from WebArena to SWE-bench and shares his 2025 predictions regarding benchmark saturation and price drops. | |
| Call to Action for Open Source Accessibility in AI | 0 | 0 | 0 | 0 | Neubig concludes his talk with an impassioned call to action for open-source AI tooling and accessible model training to prevent consolidation of power. | |
| Q&A: Benchmark Data Contamination and Help-Seeking Behavior | 2 | 5 | 0 | 0 | An audience member asks why SWE-bench scores diverge from real-world performance; Neubig educates the room on benchmark training data leakage and poor agent help-seeking calibration. | |
| Q&A: Web Navigation Modalities and Next-Gen Coding Benchmarks | 2 | 4 | 0 | 0 | Neubig answers a question from a Bing engineer regarding vision versus HTML parsing accessibility trees and markdown indices for agent web navigation. | |
| Q&A: Micro-Agents Modular Prompting and Perspectives on MCP | 2 | 5 | 3 | 0 | Neubig explains OpenHands' micro-agent prompt architecture and expresses clear skepticism toward Anthropic's MCP standard, questioning why new protocols are needed when mature APIs already exist. | |
| Q&A: Analyzing Agent Failure Modes and Debugging Scaffolding | 1 | 4 | 0 | 0 | Neubig describes the primary failure mode of agents as premature action without thorough preliminary information gathering, sharing a concrete debugging story. | |
| Q&A: Implementing Self-Improving Memory and Skill Libraries | 4 | 4 | 2 | 1 | An audience member compares OpenHands to Voyager's skill libraries and Devin's playbooks. Neubig clarifies how Agent Workflow Memory operates and why RAG between natural language and code remains difficult. | |
| Q&A: Security and Credential Management for Agents | 1 | 4 | 0 | 0 | Neubig discusses security and credentials, recommending GitHub fine-grained personal access tokens while noting that broader web infrastructure is unprepared for autonomous agents. |
Statements from this episode (25)
Neubig: Browsers, Terminals, And Code Editors Constitute The Core Agent Toolset
“Let's say I gave you a web browser and a terminal or a file system and the ability to edit text or code. What could you do with that? Everything. Yeah, probably a lot of things. This is like 99% of my, you know, daily daily life, I guess when I'm working. So I…”
Neubig Uses AI Coding Agents Five To Ten Times Daily
“I use coding agents maybe five to 10 times a day. To help me solve my own problems.”
Neubig: Arbitrary Python Execution Beats Individual Agent Tool Calls
“And the method that we adopt in open hands instead is we provide these tools, but we provide them by just giving a coding agent the ability to call arbitrary Python code. And in the arbitrary Python code, it can call these tools. We expose these tools as APIs …”
Neubig: OpenHands Equips Its Agent With Only Five Or Six Tools
“We're kind of extreme. And we're only giving the agent five tools or maybe six tools.”
Neubig: Coding Agents Excel At Using The GitHub API Directly
“The agents are super good at using the GitHub API also, so they can do, you know, things on GitHub, like finding all of the, you know, comments on your issues or checking GitHub actions and stuff.”
Neubig: Nobody Has A Good Answer For Human-Agent Interface Design
“I don't think anybody has a good answer to this, and I don't think we have a good answer to this”
Neubig: GPT Loops On Errors While Claude Tries New Approaches
“So, like, GPT doesn't have very good air recovery ability. And so, because of this, it will go into loops and do the same thing over and over and over again, whereas Claude does not do this.”
Neubig: Claude Is The Best Agent Model, Open Models Lag Behind
“I still am under the impression that Claude is the best. The other closed models are, you know, not quite as good, and then the open models are a little bit behind that.”
Neubig: OpenHands Uses A Single Prompt With No Multi-Agent Systems
“So in open hands, we do very light planning. We have a single prompt. We don't have any multi agent systems.”
Neubig: Single Agents Adapt Better Than Rigid Multi-Agent Systems
“If you have a really, really good instruction following agent it will follow the instructions as long as things are working according to your plan, but let's say you need to deviate from your plan, you still have the flexibility to do this, and if you do expli…”
Neubig: Agent Workflow Memory Boosts WebArena Performance By 22.5 Percent
“And this we demonstrated that this leads to a 22.5% increase on web arena After 40 examples.”
Neubig: All Hands AI Is Releasing A Unified Agent Benchmark
“We don't have benchmarks that test whether agents can code and do web navigation. But we're working on that and hoping to release something in the next week or two.”
Neubig: Every Major LLM Trainer Will Focus On Agents By Mid-2025
“My prediction is every large LM trainer will be focusing on training models as agents. So every large language model will be a better agent model. By mid 20, 25.”
Neubig: High Agent Running Costs Will Plunge Within Six Months
“Right now, actually, agents are somewhat expensive to run in some cases, but I expect that that won't last six months. I bet we'll have much better agent models in six months.”
Neubig: Agents Solve 80 To 90 Percent Of Tasks With Feedback
“Actual ability of models is maybe closer to 30 to 40%. So 30 to 40% of the things that I want an agent to solve on my own repos, it just solves without any human intervention. 80 to 90% it can solve without me opening an IDE, but I need to give it feedback.”
Neubig: Web Agents Achieve Much Higher Accuracy Using APIs Over UIs
“We had a paper on API based web agents. And basically what we showed is if you take a web agent and the agent interacts not with a website, but with APIs, the accuracy goes way up just because APIs are way easier to interact with.”
Neubig: SWE-bench Scores Are Inflated By Training Data Contamination
“Sweebench is on popular open source repos and all of these popular open source repos were included in the training data for all of the language models. And so, the language models already know these repos. In some cases, the language models already know the in…”
Neubig: AI Agents Including Claude Are Ineffective At Asking For Help
“I think it, my impression is that agents are not very good at asking for help, even Claude. So like when they ask for help, they'll ask for help when they don't need it and then won't ask for help when they do need it.”
Neubig: AI Models Are Poor At Pixel-Based Web Navigation
“The first way is this, the simplest way and the newest way, but it doesn't work very well, which is you take a screenshot of the website and then you click on a particular pixel value on the website and like models are not very good at that at the moment.
Like…”
Neubig: Hybrid Screenshot And Text Summaries Work Best For Web Agents
“And then the third way is kind of a hybrid where you present the screenshot, but you also present like a textual summary of the output.
And that's the one that I think will probably work best.”
Neubig: Anthropic's MCP Duplicates Existing APIs With Little Added Value
“We already have an API for GitHub. So why do we need an MCP for GitHub, right? You know, like GitHub has an API. The GitHub API is evolving. We can look up the GitHub API documentation. So it seems like kind of duplicated a little bit. And also they have a set…”
Neubig: Coding Agents Should Use Standard Developer Tools, Not Custom Protocols
“We're already developing things for programmers, you know, How is an agent different from a programmer? And it is different, obviously, you know, like agents are different from programmers, but they're not that different at this point, so we can kind of intera…”
Neubig: Inadequate Information Gathering Is The Biggest Agent Failure Mode
“So I think actually probably the biggest thing that it fails at is. Or that our agent plus Claude fails at is insufficient information gathering before trying to solve the task, and so if you provide all, if you provide instructions that it should do informati…”
Neubig: RAG From Natural Language To Code Does Not Work Well
“At least in my experience, rag from language to code doesn't work super well.”
Neubig: Most APIs Lack The Fine-Grained Authentication Needed For Agents
“For other things, they're totally not prepared to give that sort of fine-grained control. Like, most APIs don't have something like a fine-grained authentication token, and that goes into my, like, comment that we're gonna need to prepare the world for agents,…”