The Exchanges

Q Dota in the beginning, and then everybody just focused on language. And I think now the pendulum is shifting back to RL. Is there anything specific, um, in the last six to 12 months that kind of made you decide, okay, now is the time to do it? Or is it just a matter of, you know, the team coming together at the right time and the market being ready?

A It was really that, so Giannis and I led a lot of the work for post-training and kind of RL check for Gemini, and Giannis being my co-founder, and when we shipped Gemini One, we just realized that the models, like, models that were basically at GPT-IV level or above, were capable enough as kind of starting points to then post-train, or, I mean, it's, I wouldn't even call it post-training, just training. This reinforcement learning. So it's really that, I don't know if you've been able, you would have been able to do the same stuff with an earlier model, like with a GPT-III or a GPT-II, but GPT-IV had this kind of base of intelligence that you could actually go from there. And by the way, this actually happened in AlphaGo as well, and some of these systems previously, where before you did reinforcement learning, they first did imitation learning on human games. And it was important that the human playing ability was sufficiently high for you to be able to then bootstrap on top of that and then train with reinforcement learning. That is, if you would have trained AlphaGo on very weak human gameplay, then for that first project, reinforcement learning wouldn't have taken you as far. Of course, AlphaGo then figured out transition to AlphaZero, where it was trained without human data whatsoever, but I think that's where the analogy breaks. I think In the era of language models, I do…

AI assessment note: “when we shipped Gemini One, we just realized that the models... were capable enough”

Q And then on the environment, you briefly mentioned the computer is the idea that, you know, you mentioned code, but do you want to be very computer use focused? You know, some people will say the browser is kind of like the new OS anyway. Uh, I'm curious your thoughts on where you want the agent to, to live in.

A Yeah, it's, it's a really good question. I mean, if you start kind of an autonomous agent company today, you might pick from one of two categories, maybe some others, but I think broadly speaking, you could either say I'm going to build browser agents. Or kind of computer use kind of agents more broadly, or build coding agents. And I think that we're, you know, our belief as a company is that the correct wedge in, the correct starting point to this entire problem is decoding agent, because it's already, you know, software engineering is already what I would call kind of ergonomic for a language model. That is to say, You have to design the problem that you're striving for to be compatible with sort of, um, what is intuitive for the intelligence that's working on it. And for humans, what's intuitive to us is, you know, using a mouse and keyboard and kind of geospatial reasoning, because that's just how we evolve. Uh, but language models never evolved. They were trained on the internet. And because they were trained on the internet, their priors for what they, it's intuitive to them is completely different for what's intuitive to us. And so A language model, for example, has no prior for a mouse movement. It, you know, it really never seen that on the internet, but it has a really strong prior for code. So out of the two categories, say web browsing and coding, coding is the only…

AI assessment note: “our belief as a company is that the correct wedge in... is decoding agent”

Q you, they had, uh, from Twitter about how ChatGPT interleaves code of text, and it was basically just imitation learning on contractor data. Is that what you're seeing? I think, like, as you mentioned, like, a lot of approaches typically end up with some version of self-play as well that scales up yourself. How, how are you approaching your data, or like, what are your strong beliefs on your data?

A Well, you kind of, yeah, you kind of have to have both. So it's, it's really important to both get the right initial data mixture for, from supervised fine-tuning, however it is that you gather it, that feeds the correct behaviors that you want to elicit out of the model. And then you kind of amplify those behaviors with reinforcement learning. So the key thing to reinforcement learning is that it only works. I mean, It can potentially work otherwise, but practically it only works when the agent has interacted with a reward, right? It's received a positive reward for what it's done. Maybe one out of 10 times, one out of 50 times, but if it's getting zero reward, then you don't really know what behavior to amplify. And so that's why it's really important in the kind of initial mixture for it to already be doing things out of the box that are somewhat sensible, even if they're unreliable.

AI assessment note: “Well, you kind of, yeah, you kind of have to have both.”

Q kind of model access. Are you going to, uh, you know, like poolside AI is in a similar sort of category of code-focused model labs, and they focus on their product being more of like a VS code extension that people can use to access their models. Like, what is the shape of the product's thinking that you have so far? You know, I know you're not launching it yet.

A Yeah, so the kinds of problems we want to solve are maybe first just thinking about what, there are multiple form factors for a coding product, and the form factor that is most common today is, you know, this kind of cruise control form factor, which is the Copilot. Github Copilot or Cursor, which are incredible products. We use them internally. We're very happy with them. But again, this is, these are products where the engineer is driving most of the work. Like, even in agent mode, really the engineer is, like, driving most of the work, stepping the agent through, and so forth, and really supervising it. Uh, what we're building is more of the autonomous vehicle. So, right, the thing that you give it a task and it takes you from point A to point B. The form factor for that is, uh, is a bit different. I mean, one way to instantiate that form factor is through an IDE. Like, that's, that's definitely something that people have been doing. But another way is to provide access to it as an API. And what I mean by API is different than an API that just streams tokens, right? Because if you need a, you need a model that's kind of coupled to a computer, right? That it's able to read, write, and run code. So it's kind of, um, an API that rather than taking in tokens and outputting tokens, it takes in a task and a code base, and it outputs, it does some work on its own, the background, a…

AI assessment note: “what we're building is more of the autonomous vehicle”

Q Like some people already knew it, but like, You're kind of opening it up to everybody. So what do you think the role of open source will be in that? It feels like this coding agent should like default try and be open source because I feel like it can compound the learnings so quickly, but I'm curious if like the external interactions kind of are negative in a way.

A Well, uh, I think the, I think open source generally plays a really important part in the AI ecosystem because It allows for a plurality of research labs to exist. It allows, right, it's kind of, if open source models didn't exist today, open weight models, then the starting point for any AI company that wants to train things would be to pre-train something. That, basically, the starting point would be, you know, like, just a massive amount of capital required to just, you know, pre-train these models. So I think that The existence of open, there are a lot of incentives, right, for open weight models to continue existing and improving because of how beneficial they are to the ecosystem. And, you know, what I'm more worried about is a company developing a really powerful coding model and then keeping it to themselves. Because, like, at that point, it's not that the model itself has to be open sourced. Just access, you have to have access to it. So you can imagine, uh, you know, the world converging on a few companies have really powerful coding models. They basically release a nerfed version of that to the public at large, and basically have a competitive advantage by having, you know, a super intelligent coding model that only they have access to. So I think it's really important to have a player or more that, uh, basically continues Releasing the product for, for others, and I…

AI assessment note: “I think open source generally plays a really important part in the AI ecosystem”

Q that you should communicate Or measure your coding agent abilities? Obviously, right now, people focus on SweetBench. Um, do you have strong feelings on that? Do you have alternatives? How will you, how will you, I guess, encourage people to think about comparing your products versus like a Devon, like a poolside, like a Magic? You know, there's a, there's a whole bunch of people all competing in this space.

A Yeah, I think it's, I think it's kind of the heart of what's going on here. It's such an important question because it's kind of asking, how do you evaluate the capabilities? And one of the core beliefs of the company is that the evals that matter most are the ones that are done in the real world. So Autonomous coding benchmarks, let's say, like Sweetbench, are useful. I'm not going to discount them. They are useful. But let's say, you know, 90% on Sweetbench could still mean something that just falls over flat within a customer setting. So, you know, I think the, it's a pretty simple answer. You have to work with users. You have to work with customers. And you have to set up evals that are representative of the kinds of problems that you're seeing kind of repeat across multiple different types of customers. And kind of more importantly, I think that it's for this reason, like, because your customers define your evaluations. I don't think that super intelligence can be built in a vacuum, because if you build it in a vacuum, you, you have no guarantees of, you basically have no guarantees if it's super intelligence. It might be super intelligence on the things that you trained it on, that you then evaluate on in a set of benchmarks, but how do you know that actually translates to the kinds of problems that people care about in the wild? Now, if we think about super intelligence …

AI assessment note: “evals that matter most are the ones that are done in the real world”