Yang: AI models falsely claim completion on impossible coding tasks
John Yang · [State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang · Dec 31, 2025 · at 10:54
SWE-bench co-creator John Yang discusses benchmark results testing whether AI coding models recognize unsolvable software tasks or refuse them.
0:00 / 0:05exact quote · 5.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think they're all, the models are all kind of attempting and saying, like, oh, I did it, you know, so maybe not great.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from John Yang
Yang: Unit tests and independent task instances limit SWE-bench's effectiveness
“I don't like unit tests as a form of verification, and I also think there's an issue with Sweepbench where all of the task instances are independent of each other.”
Yang: Models should pass completion benchmarks before expensive multi-turn agentic evals
“Like, you know, you can do well on those first, and then sort of graduate to the multi-turn expensive stuff.”
Yang: TerminalBench enables more creative eval design than SWE-bench
“As we bench, you're confined in some sense to the domain of issues and PRs that already exist which I think has its benefits of being close to reality and natural, but I think with Terminal Bench, there's a lot of creativity that you can infuse into that”
Yang: Evaluating Human-AI Code Interaction Requires Compelling Products or Simulators
“From an academic standpoint, it feels like there's two difficult approaches to resolving that. Either you build, like, a really compelling product, like Elmarina, that people have people use consistently, which is, I mean, really tricky in and of itself. Or yo…”
Yang: SWE-bench adoption took off only after Cognition's Devin launch
“You know, we put it out October, 22, 23, and then people didn't really touch it too much. And then, of course, like, Cognition came on the scene, and Devon was an amazing release, and I think after that, it kind of kicked off the arms.”