The Exchanges

Q it's not just evaluations. I mean, evaluations, you know, you mentioned your own internal evals, but also, um, do you, do you just run a battery of tests every single time? Do you have a well-defined process? Do you have tooling that you, that you built that might be interesting to dig into, right? Like I think as a founder of any, any AI company, basically every founder needs this.

A Yeah, absolutely. So I would say the process that I like to follow is in the beginning when developing a feature, come up with a curated set of samples. Could be as small as 10, 10 samples that you run through, and then yeah, it's all notebooks basically. You run through the samples. The data set is small enough that you can keep it in your head. You become kind of very familiar with those samples, and you see how well iterations improve or regress on them beyond a certain point that's not good enough anymore, and then you go to Hundreds or sometimes maybe thousands or more of samples. So we have infrastructure for doing these kinds of evaluations, both with and without code execution. So code execution, like checking the correctness of solutions and running tests automatically can help, although not, although you can get a lot of mileage out of evals that don't have code execution in them that just compare against ground truth. The other thing that we found to be very useful is there is often a gap between what you do in research And what you have in production, those systems are not always the same. And so we've worked to, to make sure that at least some of our evaluations that we use in research, we can also run the production system against them. And so whenever we deploy a new model, if it's one of our own models, we can, we'll automatically run those evals and then we wil…

AI assessment note: “So I would say the process that I like to follow is in the beginning”

Q Yeah. Yeah. I think the other comparisons also would be, you said magic, like there's magic.dev and poolside, poolside also trying to go to market with, with extensions. But I guess those are training proprietary models, right? They're, they're investing a lot of money in that, and you've just decided to do no custom models here.

A Yes, I would say for now. So for us, there's always the question of for every feature, do we use an off-the-shelf model or an external model, or do we train our own? I think for now, for agents, uh, except for the code-based understanding, we, we, we outsource the models. However, we believe that the cost of these, or let's say, we believe that the usage of agents is going to explode. And as a result, the cost is going to explode as well. And so We feel that there is room here for training custom, custom models to help with that. But yes, for now, building a product and going to market quickly, this is what we've prioritized. And there's clearly a lot of, a lot of demand for these things out there. So we're pretty happy with that decision.

AI assessment note: “Yes, I would say for now. So for us, there's always the question”

Q Like even the, you know, when you mentioned in the, I'm reading from the blog post, you decided not to investigate this direction because of significant extra costs. Like if it costs 10 times more, how much more of a result would we get? You know, like if the, if the models start going down in costs, are we just going to do more of that?

A Yes, I expect we will. I think it will become less of over time as cost will come down. I expect it will become less of a cost question and more of a UX question because with ensembling, well, what we found is that users really want to see what the agent is doing and follow along. And if you're doing ensembling, you don't have one trajectory anymore. You have several. And then the question of how do users supervise that? As it's happening becomes pretty tricky. Like you can present the results after it's done, but typically users don't, don't like waiting that long to see what's happening. They want to make sure the agent is on the right track. So I think even if the cost ends up making sense and it might, the UX will become a problem in terms of how much more you can squeeze out of this. Yeah, I would guess you can get a few more percentage points, at least with better ensembling, because what we did is very simple. And I know others have been experimenting with this as well. I don't think it's It's, ensembling is usually not the kind of thing that just solves a benchmark, just also in other domains, you typically can get like, yeah, and this, and first we bench probably a few percentage points is my guess.

AI assessment note: “Yes, I expect we will. I think it will become less of over time”

Q What's the plan for moving it out of the ID? Do you also see a world in which just in linear, you can say, please implement, and then you don't have to actually open VS code and be on top of it?

A Yeah. So that's, that's a great question that we, we ask ourselves almost on a weekly basis. These things are getting powerful quickly. Um, and there are like, if you're doing zero to one development, It seems it's already at a point where you don't need a lot of the ID or maybe not even an ID at all to get to something that works and iterating on it. What we found so far is that when you're working in complex code bases, it's still quite often that you need to use the ID to make changes. And so for now, our product is going to be in the ID because of this reason. And also when we add support for multiple agents, we expect that it's still going to be in there. And this is also based on, on feedback from early users. I think in the future, at some point, my guess is that the IDE is going to become less of the focal point and more like an app that you can launch when you need to dig in deeper, but you spend most of your time away from it. So maybe 80% of your time is in a website or an app where you control the agents and then 20% of your time, maybe you go into the IDE. But I feel like for, for the developers that we're targeting, we're not Quite there yet.

AI assessment note: “for now, our product is going to be in the ID because of this reason.”

Q And then are something that you tried doing a more multi-agent approach with like the orientation agent and the regression fixing agent. Can you maybe talk people through that and what results you had?

A Yeah. So we found for agents that orientation in particular is extremely important for getting good value out of the agents. And so we, I believe we have this orientation phase where we try to, to look around the code base, and this is something we're also building into the product. Look around the code base and try to understand the conventions. For Sweebench, this is mostly around testing and how you execute the test. That's the main thing. But for the product, there's, there is more than that. There is try to figure out testing, try to figure out what frameworks are being used. If we can figure out the versions that are being used, all those things help give users a much better out of the box experience when they use agents.

AI assessment note: “we found for agents that orientation in particular is extremely important”

Q as a general Gemini Google revival? You know, you're on the, the Palm II team. What seems to be Uh, what, what, what should people view the Gemini progress as, right? Like, you know, 2.5 pro is out and, you know, by some accounts, it's one of the best models in the world. It is the best model in the world. Uh, yeah. What do you, what do you think?

A Yeah. It's been pretty interesting to see how, how this all developed. I think when I, uh, when I left Google for Augment, so actually the week I gave notice at Google was the week ChatGPT came out. And so this was certainly, I mean, well nominated. Not related. No, no. Everything happened before, uh, everything happened before unrelated to ChatGPT. I was actually at a conference then. I think it was NeurIps. I don't really, I'm not really sure. ChatGPT came out. Everybody was talking about it. It just all happened to coincide. And I think it's well known that this was kind of a crisis moment for Google. From what I've hear, from what I hear from friends, it seems, I mean, they're obviously taking it super seriously. And the, the, the culture has really shifted internally to be a lot more startup, be fast moving. Aggressive, and at least folks I talk to are optimistic about Google's chances to, to win this. Lots of good people there, very motivated, clearly are making like solid progress on the models. It's still a race, but I think the, the, the sentiment that I'm hearing seems to be pretty, pretty positive, I would say.

AI assessment note: “at least folks I talk to are optimistic about Google's chances to, to win this”