The Exchanges

Q And as a quick segue on that note, like, it was super interesting to see the Nobel Prizes a few months ago, or maybe that was last year at this point, like, everything converting towards AI. Do you think AI is eating all those, uh, other scientific fields?

A I think it's augmenting, like it's, um, and in some sense, I think that part of what happened was that AI's impact in the world had clearly become, ah, hard to ignore, but there's no Nobel Prize for computer science, which is a field that, you know, so, yeah, the Turing Awards have gone to AI for, you know, a number of years now, and so I think that the Nobel Committee, I'd imagine, felt like it needed to somehow Shoehorn this, uh, and so obviously there were like very impactful, uh, AI breakthroughs with AlphaFold that, um, resulted in a prize. Um, but what's interesting is that the Physics Nobel Prize was given to something that has not really had that much impact in physics, but it is, um, but I still buy it because it's kind of, uh, there's a physics smell to the breakthroughs that led to, uh, you know, these systems like called Boltzmann machines and Hopfield networks that, uh, That Jeff Hinton and Hopfield got the prize for. Um, they're very physics-y, and they, they, they look, they look like the same exact objects that physicists, physicists, physicists study.

AI assessment note: “I think it's augmenting, like it's, um, and in some sense”

Q And the next step after that was DeepMind?

A And then I went to DeepMind. At the time, I was really interested in- In Toronto? Yeah, Toronto and New York. Um, so I joined the group of, um, this researcher named Vlad Mani, who was, uh, largely credited with starting the field of DeepRL. He was the first author of the DeepQ Networks paper, which was the paper that got, um, neural networks to play Atari. And he, like, his first set of papers actually largely defined deep reinforcement learning as, uh, as a field then. Were basically DeepMind's Claim to fame for a very long time. Uh, and so I joined this group to study the problem of what we called, I mean, we started, we, the team we built together is called the general agents team, and so the whole point was to do research that enabled, you know, for us to figure out how do we build general agents. I think that, um, it was much more opaque, I think, then than now, and the big problem we were trying to solve is, um, what People called, and still do, but it, uh, called unsupervised reinforcement learning, which is really, how do you train reinforcement learning systems that are capable of, um, assigning their own rewards? Like if you don't have rewards without supervision, like in the same way that, you know, you can give things some rewards, but kids and animals, when you look at them, they, they learn a lot in an unsupervised way. Like they interact with their environments …

AI assessment note: “And then I went to DeepMind.”

Q So would coding be the brain and then everything else becomes the, uh, legs, uh, And the hands that are connected to the brain, or is that the wrong analogy?

A Yes, it's kind of like, you can kind of think of the model is the brain, and then what people call a scaffolding, agent scaffolding. That's sort of the affordances, right, the things that you can actually do. All these affordances for software, at least, you know, for digital intelligence, are going to be primarily through code. That is, the other option is teaching a model how to drag a mouse around, and it's called computer control. Some of that will happen, but I just don't think that that's going to be the majority in which, um, you know, A language model interacts with software. So if you solve coding, you've just solved how does a language model, how should it interact with software?

AI assessment note: “you can kind of think of the model is the brain”

Q As a return of, uh, reinforcement learning at the end that you described, does, does that obviate the need for RLHF? Was that a temporary solution or are those parallel Capabilities.

A I think they're parallel capabilities because they do different things. RLHF was more, the design there was more to align a language model, like a, a pre-trained language model. Um, if you play with one of these base models before they're aligned, they're really useless. They're, they are, they feel like stochastic parrots. Um, they're, yeah, they don't follow instructions. They're extremely kind of, um, high entropy, right? And The fact that you could just align them, tweak them, to align them with something that is human consumable, I think was a pretty big breakthrough, and that was RLHF. Whereas RL with reasoning is more RL to drive, kind of, um, intelligence capabilities, which is, you know, make it really good at coding, make it really good at math, make it really good at whatever target domain that you have rewards for. And you use these in tandem, where, like, RL in kind of the reasoning phase, it sort of really expands on a capability. Um, and then RLHF aligns it to be kind of human consumable. And, but they're the same thing. It's a, it's, in fact, I would say it's just, it's just the same thing. Like, the machinery is the same.

AI assessment note: “I think they're parallel capabilities because they do different things.”

Q Double clicking on the product today, what is the reality of, um, something like an autonomous coding agent? Like, how good is the state of the art right now versus what hopefully it will be in the, in the future? Like, are we in the, You know, teens in terms of like SWE benchmark. Are we higher than that for certain tasks? Where, where does it all land currently?

A On the benchmarks, this kind of quick side comment that I think that Sweebench is either, I mean, I'd call it, I mean, it's gonna get, it's getting to saturation, and it's funny that even though you have, like, these numbers, like, 70% or something like that on Sweebench, those coding models are good, but they're not solving 70% of engineering tasks. Um, so there's a sort of benchmark to real-world problem misalignment, which is always going to be the case when your benchmark is not The actual thing that customers are using it for. But in terms of progress, I've been, if there's one thing I had fairly aggressive timelines on progress in my mind, and I would say things have moved faster, even than I, than I've expected. I think that, you know, we've gone from autocomplete engines to things that are kind of semi-autonomous to now things that for junior tasks, they, they can just do them autonomously. Like it's pretty, pretty incredible. There is in some sense, we are probably at a L four kind of junior, junior engineer level of autonomy, which is Pretty incredible.

AI assessment note: “we are probably at a L four kind of junior, junior engineer level of autonomy”

Q Why is it primitive? Because it's just search-oriented? It's just a query, or?

A It's more, so there are a few things about it, but the first thing is that it's kind of, um, what we call sparse, because it's, uh, the stuff that it grabs from, you know, a large codebase has a lot of false negatives and false positives, so it's kind of, um, and it typically only does it once, right? It'll grab it, and then that's all you have, and most likely, for any meaningful query, Um, it will not have given you the information that you need to actually go do the task. So rag agents are actually, are pretty weak. What's happening now is that there's kind of a new kind of, um, retrieval that I think people are calling agentic search, which is more what Cloud Code does, and it's kind of, um, an agent that uses the, uh, same kind of command line that a human does, and goes and kind of looks for files and, uh, you know, uses the same commands that an engineer does to go look for files and, uh, You know, search for keywords and grab stuff, and it does this agentically, so as opposed to it just being a one-step process where it just, uh, pulls a bunch of embeddings in, it will use a file search tool, and then it'll look at the stuff that it did, and think about it, and then use another tool, and so forth.

AI assessment note: “it's kind of, um, what we call sparse, because it's”

Q Directionally, why and how are they able to, to do that?

A This is not unique to us. I think that when I look at how state-of-the-art agents are being designed, they're being designed very much in kind of, um, as these multi-agent systems. Agent design is effectively, it's a big reasoning agent, and that's, I mean, I think that's pretty standard, but it, uh, dispatches, like, small, long-context reasoning agents to go and kind of go search for different parts of, kind of, relevant chunks of information in the code, and so I think that it's this kind of decoupling of, uh, you know, a big reasoning agent, like maybe a smaller context, with a bunch of Um, like these little retriever scout agents. It's a different design than, uh, what's happened to date though. I'm sure that other companies will converge on it as well. Um, you really want to design your agents for the problems that you're trying to solve. So if you want like a really snappy agent that's going to answer things immediately, then this is probably not the best design for it, right? You might want something that does brag, which is very fast, um, or like a very basic search agent, kind of like what cloud code or Cursor might do, which is, uh, something that just uses the terminal and like the file system there and use the same commands as an engineer. Um, that's a lot snappier than sending a lot of these retrievers out. Um, so I think what we'll start seeing is this is, this i…

AI assessment note: “they're being designed very much in kind of, um, as these multi-agent systems.”

Q So, so coming back, I really like that term of organizational superintelligence. It does, does sound more tractable in your analogy of, uh, what a senior person would know, uh, especially, I assume, a, a senior person that actually does the work, right? Because typically, the most senior people in an organization are actually disconnected from the reality of what happens in the, in the troops.

A Yeah, I think it's more, yeah, the, the person who, there's, On every team, there's a go-to seasoned, experienced person who's very much in the weeds and understands everything. Getting systems first that are like that, and then, um, they can hold a lot more context in their heads and sort of be even better than those, you know, critical, um, you know, kind of members of the team. Uh, I think that's kind of like an oracle for an organization. That's what a super intelligence looks like, because once you deeply understand the problems that you need to solve, um, in any discipline in enterprise, Acting upon them to actually go and solve it is the easy part. And so we're really focused on coding to start, uh, for a number of reasons. We kind of think you solve this problem. It sort of solves the more general problem.

AI assessment note: “On every team, there's a go-to seasoned, experienced person who's very much in the weeds”

Q And when was the decision to leave and start a, start a company, and what was the thinking?

A Well, we shipped Gemini one and 1.5, and we realized that language models cross this threshold of utility where they're no longer research objects, they're going to be very useful. Uh, this was early 2024. Realized that the ingredients were, you know, Kind of in place to build a super intelligence. Like we, we felt that everything was there. There was one more piece to solve of going from RLHF to making reinforcement learning work. And that basically happened right over the last year with reasoning models. Um, so we felt that that would happen. Then the question was super intelligence for what? That was basically, you know, we, we felt that you can't answer this question in the abstract by Kind of being a researcher that's really far away from product and customers. You kind of had to, you really had to go in and define what that means from a product vision and, uh, what problem you're trying to solve perspective that it's like, we're not interested in building a super intelligence that will be super intelligence and mathematical Olympians. And the difference between this era of reinforcement learning and the previous era of pre-training is that when you did pre-training, you made the models kind of generally better at everything. Reinforcement learning Is much more jagged, right? It's sort of, it makes them good at what you want them to be good at. So just because you made the…

AI assessment note: “this was early 2024. Realized that the ingredients were, you know, Kind of in place”

Q On the product front, um, again, to the discussion about product versus, uh, research, uh, you know, you guys are super deep, um, PhD-type, um, world-class AI researchers, uh, but in a context precisely where you want to build product, was it part of the core team to bring in people that would bring product, or, or, or how did you think about it?

A Yeah, we've hired out, uh, I guess the, The company is, we kind of think about half product, half research, and so, um, we've, we have a, we've hired out a research team, we've hired out a product team, and then there's, um, I'd say the majority of the makeup of the company is probably, um, two-thirds of it is people who have research backgrounds at, you know, some of the big labs. Of those people, a bunch of them are kind of in this, um, in this role that is between research and product. Um, that, for example, like, the design of, like, the agent design research, um, that's a very between, like, research and product, or evaluations, like, what are you evaluating your models to be good at? Um, that typically is something that's just on research. For us, it's kind of a cross-functional end-to-end thing. The data that you're generating, the synthetic data that you're generating to train your models, um, that also cuts across all those things. So in a sense, it's, uh, I think it attracts Well, maybe people similar to Yanis and myself that came into this and just wanted to be, we just did not want to maximize another academic benchmark. We just wanted to solve real problems and have real evaluations. And so for those people, this ends up being a really good place. I think for people who would much rather kind of sit in a known entity and, um, really focus on some specific piece of …

AI assessment note: “we kind of think about half product, half research”

Q that idea has been that, um, you know, it's intelligence, building intelligence, uh, doing it automatically, which, uh, I think feels like a different avenue, but what you're saying here, or maybe not, but I think, uh, but what you're saying here is more, uh, that it's such a complex problem that if you solve that problem, then you're there. I mean, just to play back, would you just say?

A Yeah, I think that the notion of, uh, In the sense in which coding is, um, ASI complete is that you can use an intelligent coding system to build another intelligent coding system. If people like to do research in the abstract, that sounds awesome. Because it's kind of like, it's like I don't have to think about the actual problem that's being solved. If I just build an intelligence that builds an intelligence, it will figure it out somehow. Um, I don't think that's how it works. I think what, I think coding intelligence that builds better coding intelligence will Make algorithms more efficient, basically. You know, and maybe it's so intelligent that it can even start making all the product decisions for you in terms of, like, what questions you should be asking users, and, um, what features you should be building in, and to me it actually seems like a much larger, more practical problem that it's, um, there's almost no, I think research without co-designing product with it is sort of, it's sort of a meaningless pursuit now. But that was only meaningful when the ingredients for how to build, uh, artificial general intelligence, or ASI, um, were not known. I think now they're known, and so.

AI assessment note: “In the sense in which coding is, um, ASI complete is that you can use”

Q And is a sort of like snappy, one-shot agent necessarily a bad thing if you direct them at small problems, and then you try to put them together, or does the sort of individual little agent need to be smarter?

A No, I think that it's a, it's a really interesting question. And this is kind of one of the things that's interesting at research now, as opposed to before, is that before it used to be just around models. Um, but now a lot of the research is in your agent design. And so what you ask is kind of an open question. You could, there could be a hybrid system that kind of routes some queries to one agent, you know, to one agentic system, and then routes like other queries to another agentic system. It could be that you figured out some elegant, Simple, simpler kind of multi-agent system that can do both things. Um, it's kind of an open research question. It's, uh, um, it's very exciting, and in some sense, it parallels a lot of the unspoken research that was happening at DeepMind and probably in OpenAI as well during, like, pre-language models. Um, for example, projects like OpenAI's Dota V or, uh, DeepMind's AlphaStar project, which, uh, trained these, um, expert-level agents to play Pretty complex video games like Starcraft and Dota. Uh, a big question there that I don't think that many people appreciated is how do you design the environment for your neural network to actually dispatch actions, right? So the most simple thing you can think of is, well, it learns to use a keyboard and mouse like a human does. Um, that turned out not to work. So when you, when you read the AlphaStar …

AI assessment note: “what you ask is kind of an open question”

Q difference in how those agents access data versus RAG, which is pretty much like straight up Search. I guess as a side question or related question, do those data sources need to have a special protocol to lend themselves to agents or like MCP style kind of, uh, infra so that, uh, the super intelligent agents that go around and, and sort of grab information everywhere can interact with them?

A It's a, it's a really good question. Uh, so there are kind of a couple, couple interesting things to unpack there. The first thing I guess I would say is that kind of going back to some It, depending on the problem you want to solve, like different, um, types of search, um, these are all, these all fall into basically search. And maybe I would say that if you want really fast, but it doesn't really matter how accurate it is, or, you know, it just needs to be some kind of ballpark accurate, but really fast. So kind of the weakest form of search. RAG is great. Then there's this more kind of agentic search that we spoke about that, uh, the agent uses tools similar to the ones available to humans on a computer. And finally, which is kind of, um, In between the spectrum of fast and, you know, so it's, it's, it's slower than RAG, but it gives you better answers, and then even slower is what I'd call neural retrieval, which is you have a really long context model, and you ask that long context model to retrieve stuff for you. You know, you feed it everything you can, maybe use multiple of them if, you know, everything doesn't fit in one, and then you ask that model to Look at what you put in its context and retrieve the relevant stuff to you. That's kind of called neural retrieval, and that is going to take the longest. Um, it's not guaranteed to be the best, but you can train it, you…

AI assessment note: “that's kind of the spectrum of, um, search capabilities that I see in today.”

Q Reinforcement learning part in Asimov. How, how does it Manifest. I mean, you guys are super world-class, uh, RL, uh, specialists, so is that, is that the whole idea? Like, it keeps, uh, sort of, uh, learning and getting sharper with every interaction?

A Let's say before language models were useful, you kind of had to be in this world where you build the best language model, and then you figure out the product. That's kind of the world we were in, and, you know, Anthropix spent a few years building the language model, and then took off with Quad III. OpenAI, you know, GPT-II is not really productizable, GPT-III Not really either. And it wasn't really until 3.5 and four that they were able to productize it effectively. We're in a different world today where, um, language models are pretty good. And so our strategy has been, you know, we build this kind of multi-agent system. Some parts are, you know, we're training, you know, models for, we kind of see blind spots from third party models. Um, and other places the third party model stays for today. Over time, we're going to kind of abstract all of it, but we're being a bit more strategic about, uh, which parts of the system, um, you need to go after as a startup. Because that's kind of, you know, what, what you have as a benefit as a startup is that you can be a lot more focused and, um, you know, on the problem at hand. Obviously the downside is that you have to be a lot more strategic about the bets that you're taking. You don't have the resources to go and train everything all at once. So you kind of have to take it one step at a time. And so, you know, in the long term, this …

AI assessment note: “in the short term, we're applying reinforcement learning to fix problems that we're seeing”