Jesse Hu: Realistic AI evaluation requires measuring multi-turn clarification, not single-turn fixes
Jesse Hu · [Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu · Oct 19, 2024 · at 40:53
Jesse Hu explains why front-end and web agent evaluation is intrinsically harder than static software benchmarks.
0:00 / 0:25exact quote · 25.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think the broader thing is that the more and more realistic you get, the more you run into sort of like multi-turn or iterative things where now the task of the AI isn't just to just, you know, extract from your brain what the problem is and kind of directly solve it. The task is like, how can it ask you clarifying questions and extract more knowledge from your brain and iterate just really, really, really hard to evaluate.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jesse Hu
Jesse Hu: SWE-bench gains over baseline GPT-4 come entirely from agent scaffolding
“The diff between that and something like Devin is all in like, sort of like the agent scaffold or the agent code, right? So that's, what's really exciting about this stuff. It shows off what you can do just from prompting and just from adding tools.”
SWE-Bench public test splits enable trivial cheating via runtime PR retrieval
“The entire test split here is public. So you can do things like just overfit to the patches in the test set. You can do things like, let me add at runtime, pull the PR and just get the answer and just use it.”
Jesse Hu: SWE-Bench trajectory requirements deter commercial agents from submitting
“They actually require you to submit these things called trajectories that are proving what you did and that you're not cheating. And a lot of this ends in controversy. A lot of this ends in people say, well, we don't want to submit at all because if we review …”
Hu: SWE-bench repos do not represent typical app development work
“And I'll say the critique here is that these are, like, really robust you know, Repos that are, have a lot of history, have a lot of people, but they don't really represent like what it is to do day-to-day work for us that are building like apps and client thi…”
Hu: Long-context accuracy degrades; RAG remains necessary for entire large codebases
“My guess would be that, like, long context works, but it's sort of a lie as far as your accuracy, and that rag matters no matter what, because even in the longest context windows, you can't fit the whole code base.”