Swyx: SWE-bench resolution rates surged from 13% to ~50% in 2024
Shawn Wang · 2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents · Jan 1, 2025 · at 1:08:29
Swyx and Alessio Fanelli review the rapid progress of frontier models on the SWE-bench software engineering benchmark throughout 2024.
0:00 / 0:04exact quote · 4.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Keep in mind, we started the year at 13%. And so now we're about 50 open hands is around there.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Shawn Wang
Google's internal compute credit market caused it to miss GPT
“This is a thing that has been tried internally within Google, and it led to Google missing GPT.”
Swix: Standalone Vector Databases Are Over as a Category
“Everyone has vector indexes now, but like, I think it's fair to say vector databases as a standalone category are over.”
Swix: Pull requests and code reviews are dying, replaced by prompt requests
“The pull request is dying. Yeah. Right? It's going to be the prompt request. And then beyond that, code review is also kind of dying”
Swix: OpenAI will issue a cryptocurrency token to fund compute
“There is still one more shoe to drop, which is the non sovereign wealth funding that open AI needs to get, which they've promised to drop by the end of this year.
And my money is on, they have to do a coin.
Like it's, I'm not a crypto guy at all, but like, y…”
Swyx: AI models hit a 2T parameter wall, won't reach 10T
“As far as everyone is concerned, Claude, you know, Opus 3.5 is not coming out. GPT 4.5 is not coming out. And Gemini two, like we don't have pro whatever we've hit that wall, whatever that wall is. Maybe I'll call it like the two trillion parameter wall. Like …”