Sweebench

“Sweebench, you're starting in the context of an entire repository. And so it adds this entirely new dimension to the problem of finding the relevant files. And, you know, this is a huge part of real engineering”

“So I think that like real tasks are going to be much more interactive with the agent rather than this kind of like one shot system.”

“So a Sweebench eval for me takes about an hour to two hours to run on like a subset of a hundred problems.”

“That, and also like just in the enterprise, the use cases are pretty different than those represented in something like Sweebench.”

“So Gemini does pretty well on Sweebench. Sometimes Gemini publishes models that win on some of those software benchmarks. Raise your hand if you're using Gemini to write code right now instead of, you know, the obvious other name competitors. No one. Like, why…”