Guo: Breaking 10% on SWE-bench went from impossible to accessible in 2024
Sarah Guo · The State of AI Startups in 2024 [LS Live @ NeurIPS] · Dec 21, 2024 · at 8:43
Sarah Guo, founder of Conviction, reflects on the dramatic improvement in software engineering benchmark performance for AI models over 2024.
0:00 / 0:13exact quote · 13.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“If you recall, like a year ago, the point of view on sweet bench was like, it was impossible to surpass. Team percent or so. And I think the whole industry now considers that if not trivial accessible.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Sarah Guo
Guo: Cheaper AI code generation will increase software volume, not replace developers
“If we take the cost of software and high quality software down two orders of magnitude, we're just gonna end up with more software in the world. We're not gonna end up with fewer people doing development.”
Guo: Incumbents lack the reasoning trace data needed for AI automation
“And then one disappointing learning that we found in our own portfolio is no one has the data we want. In many cases, right? So imagine you are trying to automate a specific type of knowledge work and what you want is the reasoning trace all of the inputs and …”
Guo: Emerging VC firms can outmaneuver incumbents with outdated mental models
“Maybe that there'd be an advantage versus the incumbent venture firms in that when the floor is lava, the dynamics of the markets change, the types of products and founders that you back change it's a lot for existing firms to ingest. And a lot of their mental…”
Guo: Test-time scaling allows AI startups to pass compute costs to customers
“I think an underappreciated impact of test time scaling is you're going to better match user value with your spend on compute. And so if you are a new company that can figure out how to make these models useful to somebody, the customer can pay for the compute…”
Guo: Seat-based SaaS incumbents face an innovator's dilemma from AI automation
“There's a really strong innovator's dilemma. If you look at the SaaS companies that are dominant, they sell by seat. And if I'm doing the work for you, I don't necessarily want to sell you seats. I might actually decrease the number of seats.”