Neubig: Single Agents Adapt Better Than Rigid Multi-Agent Systems
Graham Neubig · Best of 2024 in Agents (from #1 on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands) · Dec 25, 2024 · at 17:34
Graham Neubig, CMU professor and co-founder of AllHands, contrasts flexible single-agent prompting with rigid multi-agent coding architectures.
0:00 / 0:19exact quote · 19.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“If you have a really, really good instruction following agent it will follow the instructions as long as things are working according to your plan, but let's say you need to deviate from your plan, you still have the flexibility to do this, and if you do explicit structure through a multi-agent system, it becomes a lot harder to do that, like you get stuck when things deviate from your plan.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Graham Neubig
Neubig: Anthropic's MCP Duplicates Existing APIs With Little Added Value
“We already have an API for GitHub. So why do we need an MCP for GitHub, right? You know, like GitHub has an API. The GitHub API is evolving. We can look up the GitHub API documentation. So it seems like kind of duplicated a little bit. And also they have a set…”
Neubig: Arbitrary Python Execution Beats Individual Agent Tool Calls
“And the method that we adopt in open hands instead is we provide these tools, but we provide them by just giving a coding agent the ability to call arbitrary Python code. And in the arbitrary Python code, it can call these tools. We expose these tools as APIs …”
Neubig: SWE-bench Scores Are Inflated By Training Data Contamination
“Sweebench is on popular open source repos and all of these popular open source repos were included in the training data for all of the language models. And so, the language models already know these repos. In some cases, the language models already know the in…”
Neubig: GPT Loops On Errors While Claude Tries New Approaches
“So, like, GPT doesn't have very good air recovery ability. And so, because of this, it will go into loops and do the same thing over and over and over again, whereas Claude does not do this.”