Hu: Coding agents produce bloated edits unless constrained by brevity priors
Jesse Hu · [Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu · Oct 19, 2024 · at 57:49
Jesse Hu discusses code quality and agent behavior on SWE-bench tasks during a Latent Space Paper Club session.
0:00 / 0:15exact quote · 15.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“There's something nuanced about this data set in particular where all the edits are super short and it's like a prior that you can put into your code. But if you don't have that, then yeah, like agents tend to just keep making obnoxiously long edits”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jesse Hu
Jesse Hu: SWE-bench gains over baseline GPT-4 come entirely from agent scaffolding
“The diff between that and something like Devin is all in like, sort of like the agent scaffold or the agent code, right? So that's, what's really exciting about this stuff. It shows off what you can do just from prompting and just from adding tools.”
SWE-Bench public test splits enable trivial cheating via runtime PR retrieval
“The entire test split here is public. So you can do things like just overfit to the patches in the test set. You can do things like, let me add at runtime, pull the PR and just get the answer and just use it.”
Jesse Hu: SWE-Bench trajectory requirements deter commercial agents from submitting
“They actually require you to submit these things called trajectories that are proving what you did and that you're not cheating. And a lot of this ends in controversy. A lot of this ends in people say, well, we don't want to submit at all because if we review …”
Hu: SWE-bench repos do not represent typical app development work
“And I'll say the critique here is that these are, like, really robust you know, Repos that are, have a lot of history, have a lot of people, but they don't really represent like what it is to do day-to-day work for us that are building like apps and client thi…”
Hu: Long-context accuracy degrades; RAG remains necessary for entire large codebases
“My guess would be that, like, long context works, but it's sort of a lie as far as your accuracy, and that rag matters no matter what, because even in the longest context windows, you can't fit the whole code base.”