Glaese: OpenAI no longer trusts further score improvements on SWE-bench Verified
Mia Glaese · The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals · Feb 23, 2026 · at 14:34
Mia Glaese (VP of Research at OpenAI) explains why SWE-bench Verified has reached saturation and is no longer an effective benchmark for frontier coding capabilities.
0:00 / 0:11exact quote · 11.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Issues with the benchmark that means that now that we're at like 80%, we don't really trust like further improvements on it, but like it does measure something that is like a real like capability of models.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Mia Glaese
Glaese: Open-source benchmarks cannot use canary strings to avoid contamination
“There's like multiple avenues, but like the problems are sourced from open source repos. So it's not just like when we usually publish evaluations, we publish evaluations, and then we add canary strings to ensure that, you know, they are easily filtered out at…”