Embiricos: Benchmark-Passing SWE Agent Outputs Are Often Unmergeable in Practice
Alexander Embiricos · ChatGPT Codex: The Missing Manual · May 16, 2025 · at 10:36
Alexander Embiricos, OpenAI engineer on ChatGPT Codex, explains why benchmark evaluations like SWE-bench fail to capture the real-world usability of generated code.
0:00 / 0:10exact quote · 10.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Because if you look at a lot of, like, Sweebench passing, like, outputs from, like, an agent, they're not really, like, PRs that you would merge, because, like, the code style might be, like, different. Like, it works, but the code style is different.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Alexander Embiricos
Embiricos: Scaffolding-heavy AI agents are limited by developers' mental capacity
“A lot of, like, agents that I see are really impressive, but it's basically, like, part of what's impressive is it's like a bunch of developers building this, like, really bespoke state machine around a bunch of, like, short model calls, and so then the upper …”
Embiricos: Majority of Future Code May Be Written by Parallel AI Agents
“In, in a future world that we imagine where actually you know, maybe the majority of code is actually being written by agents that we're delegating to, you know, doing tasks in parallel. It becomes, like, critically important that you can actually, like, integ…”
Embiricos: Specialized domain training yields outsized returns in general models
“If you can, like, build, do something very specific for, like, a specific purpose, actually, when you bring that and you bring it into the generalized model, like, you might even get outsized returns on that. Because there's, like, transfer from all these diff…”
Embiricos: OpenAI open-sourced Codex CLI to standardize agent safety
“Part of why we made the Codex CLI open source is, like, a lot of problems, like, safety issues that you need to figure out for how to deploy these things safely, and no one should have to figure these out, like, more than once. So, that's why we went for, like…”
Embiricos: The best Codex users spend 30 seconds max prompt crafting
“The way we see people who, like, love Codex the most using it is they don't, they think for, like, maybe 30 seconds max about their prompt. It's just like, oh, I have this idea, like, boom. Oh, like, there's this thing I wanna do, like, boom. Oh, like, I just …”
Embiricos: Clean architecture is more important than ever with AI agents
“Good architecture is like even more important than ever. And like, I guess the fun thing is like for now, that's something that humans are really good at. So like, you know, kind of good, you know, important for the software engineers to do their job.”