Oct 19, 2024 · 1h 1m · latent-space
[Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this Paper Club presentation, Jesse Hu and host Eugene explore the architecture, evolution, and practical realities of SWE-bench, SWE-bench Verified, SWE-bench Multimodal, and MLE-bench. They analyze how multi-step agent scaffolding, human verification pipelines, and structured retrieval strategies bridge the gap between frontier language models and complex software engineering tasks.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 2.7 Guest teaching 3.9 Guest disagreement 0.3 The hosts pushing back 1.2
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 52:58 Disputing Kaggle obfuscation robustness
Eugene challenges Jesse's point about task obfuscation, emphasizing that memorized code on GitHub nullifies text-masking defenses.
Hardest push from the hosts ▶ 52:58 Pressing on training data contaminationEugene directly reframes the overfitting problem from task comprehension to underlying code memorization across open-source repositories.
Biggest teaching moment ▶ 21:10 Masterclass on human annotator calibrationA participant provides deep domain knowledge on inter-rater reliability, forced-choice rating scales, and high-severity ensembling.
The host holds their own ▶ 39:40 Firsthand UI testing founder experienceEugene draws on his background founding a UI testing company to explain why subjective visual testing fails across varied screen resolutions.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Technical Setup and Screen Sharing Confirmation | 1 | 2 | 0 | 0 | Jesse starts off the session introducing the purpose of SWE-bench compared to HumanEval. Eugene facilitates the tech setup without confrontation. | |
| Mining Pull Requests and Constructing SWE-bench | 0 | 4 | 0 | 0 | Jesse gives an uninterrupted overview of how SWE-bench mines GitHub pull requests and test patches across popular Python repositories. | |
| Explaining Fail-to-Pass and Pass-to-Pass Unit Tests | 3 | 4 | 0 | 1 | Eugene asks for clarification regarding how unit tests in pull requests are isolated and evaluated. Jesse breaks down the distinction between fail-to-pass and pass-to-pass tests. | |
| Deconstructing Benchmarks and Comparing OpenAI o1 | 4 | 4 | 1 | 3 | Eugene relays audience questions comparing SWE-bench to OpenAI o1 and chain-of-thought methods. Jesse clarifies the distinction between agent scaffolds and raw model capabilities. | |
| Environment Setup Challenges and Cheating Vulnerabilities | 0 | 5 | 0 | 0 | Jesse details the practical engineering pain points of reproducing environments via Docker and the vulnerabilities that allow agents to cheat by leaking future information. | |
| Introducing SWE-bench Verified and Solvability Criteria | 0 | 5 | 0 | 0 | Jesse covers the motivation behind OpenAI SWE-bench Verified, explaining how human annotators evaluated issue solvability and filtered ambiguous tasks. | |
| Human Annotation Pipeline and Quality Filtering | 7 | 3 | 1 | 2 | A participant chimes in with deep practical experience on human annotation calibration, non-neutral 4-point scales, and conservative ensembling methods. | |
| Human Demonstrations vs LLM Verification and CriticGPT | 6 | 4 | 0 | 2 | The participant references OpenAI CriticGPT and context-length degradation curves, sparking a technical discussion on retrieval vs long-context LLMs. | |
| Analyzing Top Agent Strategies from Execution Trajectories | 2 | 5 | 0 | 0 | Jesse shares practical insights derived from inspecting agent trajectories in the benchmark experiments repo, contrasting Agentless with Honeycomb and Guru. | |
| SWE-bench Multimodal for JavaScript and UI Tasks | 0 | 5 | 0 | 0 | Jesse reviews the SWE-bench Multimodal paper, highlighting the shift to JavaScript/TypeScript and the strict requirement for visual assets in problem statements. | |
| Challenges in Objective UI and Web Agent Evaluation | 7 | 2 | 1 | 4 | Eugene steps in with domain authority from running a UI testing startup, highlighting human unreliability and multi-screen PSD translation subjectivity. | |
| Introducing MLE-bench and Autonomous Kaggle Agents | 0 | 5 | 0 | 0 | Jesse outlines MLE-bench by OpenAI, detailing how agents are evaluated on end-to-end Kaggle competitions with standard compute budgets and submission APIs. | |
| Practical Limitations, Costs, and Real-World ML Disconnect | 1 | 5 | 0 | 0 | Jesse performs cost math on MLE-bench runs and critiques the gap between clean Kaggle competition definitions and messy real-world data science workflows. | |
| Evaluating Kaggle Data Contamination and Overfitting | 6 | 3 | 2 | 5 | Eugene pushes back regarding Kaggle memorization, arguing that masking task descriptions fails if the model has memorized task-to-code implementations from GitHub. | |
| Code Smells and Solution Quality in SWE-bench | 4 | 3 | 0 | 1 | Audience members ask about generated code maintainability and AI code smells before Eugene concludes the session with announcements for upcoming paper reviews. |
Statements from this episode (21)
Jesse Hu: Frontier models have effectively capped out HumanEval performance
“I think through multiple techniques and through the most recent models, you can actually basically cap out on the performance on human eval.”
Hu: SWE-bench repos do not represent typical app development work
“And I'll say the critique here is that these are, like, really robust you know, Repos that are, have a lot of history, have a lot of people, but they don't really represent like what it is to do day-to-day work for us that are building like apps and client thi…”
Hu: SWE-bench scores jumped from 3% on GPT-4 RAG to 43%
“If we just use GPT-IV plus RAG, what do we get? It's, like, a measly three percent. And then up to the most recent submissions where they get up to 43%.”
Jesse Hu: SWE-bench gains over baseline GPT-4 come entirely from agent scaffolding
“The diff between that and something like Devin is all in like, sort of like the agent scaffold or the agent code, right? So that's, what's really exciting about this stuff. It shows off what you can do just from prompting and just from adding tools.”
Jesse Hu: SWE-bench competitors use identical base model APIs, differentiating via prompting
“Everyone is using the same base models here, or the same model APIs period. I don't know if anyone except for cosine was even training, but like the diff here is all in prompting.”
SWE-Bench public test splits enable trivial cheating via runtime PR retrieval
“The entire test split here is public. So you can do things like just overfit to the patches in the test set. You can do things like, let me add at runtime, pull the PR and just get the answer and just use it.”
Jesse Hu: SWE-Bench trajectory requirements deter commercial agents from submitting
“They actually require you to submit these things called trajectories that are proving what you did and that you're not cheating. And a lot of this ends in controversy. A lot of this ends in people say, well, we don't want to submit at all because if we review …”
Hu: Some SWE-bench Tasks Are Unsolvable Due to Missing Context
“Sometimes it feels like you try to read the issue. And you're just like, okay, even if I was like an Oracle or like some sort of God coder, I couldn't, I would not be able to solve this because there's some context that was left out here. There's something tha…”
Hu: AI Models Should Eventually Reach 100% on SWE-bench Verified
“And in that way, I think we should be able to hit up a hundred percent eventually.”
Hu: Long-context accuracy degrades; RAG remains necessary for entire large codebases
“My guess would be that, like, long context works, but it's sort of a lie as far as your accuracy, and that rag matters no matter what, because even in the longest context windows, you can't fit the whole code base.”
Hu: Honeycomb agent framework ranks number one on full SWE-bench dataset
“And then there's another one called Honeycomb, which also scored really well, and I think is number one on the full set.”
SWE-bench Multimodal paper baseline scores 12 percent
“They want to show off that this is, you know, guys, this is really hard. It's a really hard benchmark, so we can only get 12% on it today. According to, you know, their implementation.”
Hu: Evaluating UI correctness in SWE-bench Multimodal is highly subjective
“Zooming out, if you're true to try to judge whether a UI is correct, it's like extremely subjective. It might be iterative. It might be, you know, I have to interact with it first to get it right. So I'm super curious to see how they actually do the judging cr…”
Jesse Hu: Realistic AI evaluation requires measuring multi-turn clarification, not single-turn fixes
“I think the broader thing is that the more and more realistic you get, the more you run into sort of like multi-turn or iterative things where now the task of the AI isn't just to just, you know, extract from your brain what the problem is and kind of directly…”
Hu: OpenAI o1-preview achieves bronze medals in 17% of MLE-bench competitions
“Their final results with a one preview and this a scaffolding from a different company was that they got a bronze medal. I don't think I've ever achieved once but I haven't competed that much in. 17% of competitions.”
Hu: OpenAI o1-preview surpasses human Kaggle Grandmasters with seven gold medals
“Since a grandmaster requires five gold medals and oh, and preview gets an average of eight or sorry, seven gold medals. They're out competing even capital grandmasters.”
Hu: AI agents lack intrinsic time awareness, failing to budget execution limits
“What's interesting is like, and I, I've seen this in practice, it's like, it's hard to get the agent to say, to think in numbers of steps, and especially in time, because it doesn't know time. So if you tell it like, please complete under 50 steps, it won't do…”
Hu: GPU setups showed virtually no agent performance gain over CPU-only
“They compared a CPU only setup to a GPU setup to a multi GPU setup, and it kind of made no difference really.”
Hu: Single MLE-bench evaluation run with OpenAI o1-preview costs $4,000
“Just for one seed, For one run of these things cost 4000 dollars all in with the GPU plus the tokens. And a bulk of the cost was actually the token, so even if you cut the GPU out, it'll still cost you three grand to run on one preview.”
Hu: MLE-bench authors found obfuscating competition details did not show overfitting
“They do a lot of checks against overfitting on the Kaggle tasks themselves, and so they do something where they obfuscate some of the details of the Of the competitions, and then they rerun it. And I guess if they were overfitting on the competitions themselve…”
Hu: Coding agents produce bloated edits unless constrained by brevity priors
“There's something nuanced about this data set in particular where all the edits are super short
and it's like a prior that you can put into your code.
But if you don't have that, then yeah, like agents tend to just keep making obnoxiously long edits”