Terminal-Bench
also referred to as: terminal bench
includes Terminal Bench 2.0, Terminal Bench 1.0, Terminal Bench 3
13 statements across 3 episodes · 9 bullish · 0 bearish · 4 people on the record · first statement Oct 18, 2025 by Mike Merrill · said 88 times in 9 episodes since 2025 · across every show →
Mentions by year, the whole family
brought up most by Mike Merrill (33), Andy Konwinski (7), Alex Shaw (6), Shawn Wang (5), John Yang (5), Ivan Burazin (1), Alex Krentsel (1), Alessio Fanelli (1)
tap a year for its mentions
2026 6 mentions in 4 episodes 2 per episode
- Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah Hill-Smith
- Exo: Harnesses should see their own code and logs — Alex Krentsel, UC Berekeley / Google Research
- AI Agents Need Computers: 74% MoM Growth, 850K/Day Runs, & New Agent Cloud — Ivan Burazin, Daytona
- Measuring Exponential Trends Rising (in AI) — Joel Becker, METR
- every mention in 2026, scene by scene →
2025 82 mentions in 5 episodes 16 per episode
- Terminal-Bench 2.0: the most impt coding agent benchmark of 2025 gets a v2! Launch + Q&A w/ founders
- Terminal-Bench: Pushing Claude Code, OpenAI Codex, Factory Droid, et al to the limits
- [State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang
- [State of Research Funding] Beyond NSF, Slingshots, Open Frontiers — Andy Konwinski, Laude Institute
- [State of AI Papers 2025] Fixing Research with Social Signals, OCR & Implementation — Team AlphaXiv
- every mention in 2025, scene by scene →
Everything said about Terminal-Bench, oldest first
Oct 18, 2025 neutral
Oct 18, 2025 positive
Alex Shaw: Base model selection matters much more than agent frameworks
“And if you look at the difference between the same model and the biggest spread across agent frameworks versus the same agent framework and the biggest spread across models, It's much larger across models, which I think goes to show that like improving models …”
Oct 18, 2025 neutral
Oct 18, 2025 positive
Merrill: AI evals will shift to observing real jobs over 2-3 years
“I think like the future of evals does look much more like this is like observing people who are doing their real jobs and then translating those real jobs into a format that allows you to evaluate language models and harnesses on them. And it's probably going …”
Oct 18, 2025 bullish
Oct 18, 2025 bullish
Merrill: AI terminal interfaces will outpace GUI computer use development
“The bet here that we made was that GUI based computer use was going to be much slower to come into its own. Than terminal based use. And there's a few reasons for this. I mean, as I mentioned earlier, text is just the modality that works best with these models…”
Oct 18, 2025 positive
Oct 18, 2025 bullish
Nov 8, 2025 positive
Shaw: Terminal-Bench accepted 89 of 250 crowdsourced tasks for co-authorship
“We told people if they created three tasks for Terminal Bench, they could be a co-author on the paper that we eventually published, and I think we got maybe 250 task contributions, and 89 of them made it into the benchmark”
Nov 8, 2025 neutral
Nov 8, 2025
Nov 8, 2025 positive
Dec 31, 2025 positive
Yang: TerminalBench enables more creative eval design than SWE-bench
“As we bench, you're confined in some sense to the domain of issues and PRs that already exist which I think has its benefits of being close to reality and natural, but I think with Terminal Bench, there's a lot of creativity that you can infuse into that”