SWE-bench tasks do not reflect actual enterprise software engineering use cases
Matan Grinberg · The AI Coding Factory · May 29, 2025 · at 26:27
Matan Grinberg, co-founder of Factory AI, discusses why Factory no longer prioritizes competing on public coding benchmarks like SWE-bench.
0:00 / 0:06exact quote · 6.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“That, and also like just in the enterprise, the use cases are pretty different than those represented in something like Sweebench.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Matan Grinberg
Optimal future AI coding interfaces will not evolve from traditional IDEs
“Our take is that it is very unlikely that the optimal UI or the optimal interaction pattern for this new software development where humans spend much less time writing code. I think it's very unlikely that that optimal interaction pattern will be found by iter…”
Massive context windows will not eliminate the need for RAG retrieval
“Like if you do have billion token context window model, you throw it all in there. It's still gonna be more expensive. The reason why retrieval is so important for us is because even if there is a model that's going to have these larger context windows, and ce…”
Grinberg: LLM coding capability directly boosts performance on all downstream tasks
“Capability in code is really core to performance on any LLM and like loosely, the better any LLM is at code, the better it is at any downstream task, even that's like writing poetry.”
Grinberg: Human-written code percentage will drop within a few years
“The reality is very clearly in the next few years, the amount of lines of code written by a human will go down. Like the percentage of code written by humans will go down.”
Grinberg: AI agents will finally make test-driven development work
“The promise of test driven development is going to finally be delivered with this world of AI agents that are working on software development”
Grinberg: Developer sentiment and timeline compression matter more than engineering metrics
“At the end of the day, no one really cares about the metrics. What people really care about is like developer sentiment. When you're kind of playing that game at the end of the day, if you want to do a metric, talk to developers and ask if they feel more produ…”