Merrill: 60% to 70% of SWE-bench Verified tasks come from Django
Mike Merrill · Terminal-Bench: Pushing Claude Code, OpenAI Codex, Factory Droid, et al to the limits · Oct 18, 2025 · at 19:06
Terminal-Bench co-creator Mike Merrill critiques the repository diversity in SWE-bench Verified compared to Terminal-Bench.
0:00 / 0:06exact quote · 6.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“If you go look at sweet bench verified, I think like 60, 70% of the tasks in there are from Django.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Mike Merrill
Merrill: AI terminal interfaces will outpace GUI computer use development
“The bet here that we made was that GUI based computer use was going to be much slower to come into its own. Than terminal based use. And there's a few reasons for this. I mean, as I mentioned earlier, text is just the modality that works best with these models…”
Merrill: AI evals will shift to observing real jobs over 2-3 years
“I think like the future of evals does look much more like this is like observing people who are doing their real jobs and then translating those real jobs into a format that allows you to evaluate language models and harnesses on them. And it's probably going …”
Merrill: Frontier AI labs will center operations around vertical products
“And with the Claude codes and the codec CLIs and the deep researchers, researchers, you starting to see some evidence that the products are going to be a much more central part of how these frontier labs operate.”
Merrill: Terminus-2 harness yields highest agent performance on Terminal-Bench 2.0
“Terminus-II, which is our harness for measuring language models, is the highest performing agent for many of these models.”