The Exchanges

Q Great. Going back to big tech, perhaps with Oracle that you just Mentioned. Isn't that kind of both a gamble, but also kind of rationale for a company like Oracle to be leveraging an old business to turn it into something new?

A Well, you know, everyone in, in, generally in a bubble, everybody's a rational actor. Almost everyone's a rational actor given their situation. If you're Sam Altman, you've got a commodity technology. You've got, you're competing with people who have giant legacy cash flows. You don't have your own infrastructure. Don't really have any differentiation, but you've got massive mindshare. So what do you do? Well, you try and swap that for hard assets, and you try and talk your way into a self-fulfilling prophecy. You try and swap that for hard assets, and you try and turn those lightly engaged nine hundred million weekly active users into something, something more tangible. If you are Larry Ellison, like, you've got this very cash generative legacy business that's been in structural decline for 25 years. Most people going through YC have never heard of Oracle, like, almost literally. Um, no one has invited you to a party since like, 1998. Um, so what do you do when here is this thing? You grab onto it with both hands and you burn your way through. The same with NVIDIA. I mean, I haven't looked at, I haven't updated my number here, but like, I think Q three last year, I think NVIDIA had something over seventy billion dollars of trading 12 months free cash flow. So they can't give the money to TSMC fast enough. TSMC won't take it fast enough, and TSMC, as it says, like, dude, this i…

AI assessment note: “you've got this very cash generative legacy business... You grab onto it with both hands”

Q What kind of agents are we talking about? Are we talking about coding agents? I think you said somewhere that every agent should be a coding agent.

A So we see a divergence between, like, two different types of agents out there. One of them is, like, conversational agents. So these would be, like, the customer support, customer experience, chatbots. These have, these, like, require really low latency. Voice is oftentimes the medium that they interact with, and that's one style of agents that are mostly, like, conversational. They don't do a ton of tool calling. They'll maybe do, like, one or two, because they can't do too many or will take too long. But then we see this other style of agents, which Sequoia came up with this name, long horizon agents, and I, and I really like that they can operate over long horizons. They can do some planning. They can maintain coherence. And yes, a lot of them end up looking like coding agents. And I think there's probably like, there's a few reasons for that, but one code is really useful. You can do, you can use code to do a bunch of different things. You can use it to parse text files. You can use it to do things programmatically. Like you want to loop over a hundred different files rather than doing a hundred different tool calls. You can write a script that does that. So code is like really generally useful. But then also the models are trained on code. And so all the big model labs have been RL and code and bash and, and, and, and editing files into those models. And so that is the stu…

AI assessment note: “we see a divergence between, like, two different types of agents out there.”

Q As you describe all of this, um, I'm trying to figure out what the concept of memory means, because it seems like there's memory in the file system, there's memory in the sub-agents. Is memory in other places as well? What is memory for agents?

A Memory is super important. I mean, I think a lot of what we've been talking about so far, I would describe as like short-term memory, which is really like within a particular thread or conversation. So even when you summarize, that's still within a particular kind of like thread. The more interesting type of memory I think is long-term memory. And so what long-term memory is, there's three different types of long-term memory. Uh, one is like semantic memory. And so that's basically, you can think of like RAG for that. So there's a lot of facts that Somehow get put into this kind of like semantic store that could be through conversation. So I talked to you, I learned things like anthropomorphizing better here, but like I talked to you, I learned things, I store them in some place and I can go back and say, oh yeah, Matt's, you know, favorite drink is whatever he's drinking at the moment or something like that. And so that's like a semantic fact that I can store that you can think of it. Yeah. Just retrieval rag, um, episodic. And we know how to do that. We know how to do rag and stuff like that. What the interesting part there is how do those things Get into memory. How did those get extracted? That's a little bit more, you know, that, that that's, that's where that's not really figured out. And there's some interesting thinking to be done there. Um, episodic is basically previo…

AI assessment note: “there's three different types of long-term memory. Uh, one is like semantic memory.”

Q Is there another part of the ecosystem that is stable enough that's worth investing into obviously as I'm Listening to you speak, it's such a dynamic field. What about MCP, for example? Has, has everybody normalized on MCP being the standard?

A Yeah, MCP is fine. I mean, it's a way, it's a way to expose APIs in a standard format. Um, it's great. It has, it has a bunch of other kind of like features like elicitation and things like that, that are not supported by nearly as many kind of like clients. I think the core part of like, how do you expose, uh, how do you expose APIs in a standard way is, is definitely useful. I mean, I think the, um, I think the stable stuff is probably stuff that's a little bit more, um, lower level. So we do a bunch with observability. I think no matter what these agents look like, you're going to want to know what's going on inside of them. Um, same with evals, no matter what they look like, you're going to want to measure them in some way. Um, sandboxes, I actually think are a really good example of this. Like they're pretty low level infrastructure piece. You know, if, if agents never write any code, then okay, maybe they're not useful, but I think it's trending where Basically all agents will write code, so that's a very interesting piece, I think. Um, those are, like, the, uh, I think, like, so I think, like, pretty clearly agents will be long-running and stateful, and so I think we have a deployments product. I think a lot of the, uh, I think deployments products that let you build long-running stateful things will be kind of, like, interesting no matter what. And that's kind of how we…

AI assessment note: “Yeah, MCP is fine. I mean, it's a way to expose APIs”

Q And do you think, uh, conversational agents become coding agents as well as they go deeper into the stack?

A This is a really good question. I mean, I, I, we talk a bunch about this internally because we're debating whether we should build like a different type of agent harness for these types of agents. I think, um, I think there will kind of be a convergence when there are agents that can reliably like kick off and manage other long horizon agents. So one of the things that we're seeing in coding is that people want this experience of being able to kick off a bunch of other, Like do a bunch of work, kick off a bunch of agents, but, but keep on chatting with like the main agent. And that's very similar to like a conversational agent in some sense, right? Like it's, it, you've got that, like, you've got that like constant kind of like back and forth, um, latency TPD, but then, you know, the, these voice agents, I think will obviously want to do more and more like long running things in the future. And I think the way that you'd do that is you'd, you'd basically have two agents, one that runs in the background and is kicked off by this other kind of like conversational agent. So it could all kind of like converge into this, into the single harness that just supports basically long running async background agents as a tool.

AI assessment note: “I think there will kind of be a convergence when there are agents”

Q So seen from LinkChain as a company, uh, LinkChain's perspective, sandboxes are something to recap that, that you call, uh, what's your surface area of, of contact with the sandbox?

A So I think there's two interesting ways that agents can use sandboxes. One, you can basically spin up the sandbox and then install the agent there. And have the agent running inside the sandbox. Another way to use sandbox is you can actually have the agent running outside and then have it call the sandbox like as a tool. Um, and in, in practice, we see people doing about fifty-fifty between each of these. I wrote a Twitter article on this and people from both sides yelled at me and were like, how can you even say there's another option? It clearly has to be X or it clearly has to be Y. So I do think it's a little bit up in the air. One thing that I'd maybe say is like, I think a lot of these agents Um, a lot of these agent harnesses are coming from the coding agent world. And if you look at like something like cloud code, it's very much built to be run on kind of like your, your local machine or your local kind of like system. And so in that, so people who are coming from the world of like, oh, I see cloud code. I'm going to take cloud code or cloud agent SDK and run it. They almost always spin up a sandbox and then install cloud code in there because that's the, that's the way it's meant to be run for people who are coming at it more fresh or holistically. And they're like, Hey, you know, I've got this agent. I want to give it coding ability. That's where we see people spinnin…

AI assessment note: “there's two interesting ways that agents can use sandboxes”

Q And, uh, to that exact point, like how, us contrast and compare, so RLVR, uh, versus, uh, which is reinforcement running with verifiable rewards against what it is that you do. Is that, are those just completely different approaches? Because they both seem at the same thing, which is to basically get to, uh, perfection.

A Yeah, I think we, um, we, the world is realizing that we need verification. Verification means very different things in math. I think in early 25 or late 24, Ah, it means the numerical answer associated with each problem. Now, the thing is, like, one is, you know, reward hacking. Like, you know, we have seen from, say, Frontier Math and other benchmark, which only compels a numerical answer that it doesn't actually necessarily reflect the model's capability in the logical reasoning. So it's able to get to the answer without reasoning through it, which is quite fascinating. I mean, there's always these, I did Math Olympia before, and there's always this, like, classmate who's really good at guessing the answer. I don't know, like, AIME, which is this exam, um, that all the answers are between zero, zero, zero to nine, nine, nine. Like, I remember there's one year where my friend told me that, like, you know, he just basically guessed three questions correctly. Whereas the rest of us need to, like, read something through, and like, Jesus Christ, he just put, like, a zero in there, and then somehow that answer is indeed zero. It sounds very unfair, and you know, in a way, in high school, the teacher will, like, ask you to show your work. So, for a while, I think, like, verifiable reward means that final numerical, like, output. I think that like people are now realizing it doesn't…

AI assessment note: “Verification means very different things... verifiable reward means that final numerical, like, output”

Q Great. Uh, and what kind of math are we talking about? Is that high school math? Is that competitive math? Or is that deep research math?

A Yeah, I think people generally start with competitive math because it's kind of, you know that you have like a non-solution, and then you, um, kind of start hill climbing the infinite, sort of like infinitely high mountain of math. Um, there are actually two axis of difficulty. One is how creative the solution is, and The other one, roughly speaking, is how abstract the mathematical object is. So say a qualifying exam can be incredibly abstract, but the sort of creativity required to solve each problem might not be that high. It might be very standard. On the other hand, IMO problem, while it's very sort of easy to understand, even by high school students, not very abstract, but it's incredibly creative.

AI assessment note: “I think people generally start with competitive math because it's kind of”

Q obviously you are a new entrant in the field of AI, where there's tremendous amounts of competition. So I think for voice in particular, like the obvious question is why has OpenAI or Google or Meta not already won voice AI? And I think you probably alluded to Some of the reasons up front, but, uh, what, why is that? Why can a small company hope to become the leader?

A So one thing I mentioned was, uh, if you have the right team, it can be extremely small and still make a significant impact. Other, uh, arguments I think is, one is focus. So, for example, if you look at large multimodal models, right? Like these generic models that understand images and can generate text and can produce code and so on. You have, uh, like a limited budget, which is the number of parameters and data you're going to feed to your model. When you want to add speech to them, you're fighting with coding and, uh, image understanding and so on. So you are playing with a lot of trade-offs that are irrelevant to the task that you want to solve. And at the same time, these models are so large, they cannot run at scale because they will just make everyone lose money in the process. The only A format that makes sense for speech models to run at scale is to be extremely compact, which also means that the training resources you need to train them are much smaller than what you need in, uh, to train other kinds of models. So the resources are not as challenging as for text models. I think also another aspect is, uh, in a way, not trying to just make a conversational product. So really making building blocks so that people can build the product. So we could make The Gradium conversational assistant and think a lot about its capabilities and what it can do and what it cannot do …

AI assessment note: “if you have the right team, it can be extremely small and still make a significant impact.”

Q You mentioned data a second ago. How does that work for voice AI, and how does that compare to text AI? Obviously, text AI, the LLMs are training on the whole internet, but presumably there's a lot less Audio and speech data to train on? How does that work?

A So if you do the math, basically like training on a few trillions of tokens, which is what you will do for, you know, like a basic text model that will amount to hundreds of millions of hours of speech or something like that, which is kind of amounts that are very hard to get. I think this is a very interesting question that comes up in a lot of discussions and everybody has their theories. In particular, one impact, uh, one, let's say, one attribute of speech data Was that if you train a conversational model on speech data, it's going to be much less intelligent than a model trained on text. And I think it's because when you listen to speech data, it's, uh, the density of information is, uh, is much lower than you would have in text. So you don't have Wikipedia, like, uh, you know, in speech data, you don't have Stack Overflow, Reddit, and so on. Getting your model to learn about the world from speech, I think it's a terrible idea. I think you should start. I mean, you know, we have text on that. So what we did for mostly is we started from the text model and then we, we, we, we, We took this text model and, and, and trained it on speech while trying to prevent as much as possible a loss of intelligence. So all the time we will recompute the text metrics and they will degrade, but we are trying, you know, like to keep it a bit contained, but indeed the, the quality of, uh, of …

AI assessment note: “density of information is, uh, is much lower than you would have in text”

Q And then there's, uh, an additional aspect to this, which is that voice can also, and should be pretty often on device versus, uh, an API call to the cloud. Is that, is that fair?

A I think what is very, very challenging right now is if you want to have the full intelligence on device, like the, like your full conversational AI on device. Honestly, I would say at this point, if you want such a model to be useful, we are not there yet, right? You can have a Something that can chit chat a bit and it will be decent. Or we also have shipped, uh, models on device, but they are much more constrained in terms of applications. So for example, we, we started a year ago with, uh, uh, on device speech to speech translation, which is something that makes a lot of sense because when you're traveling, maybe, you know, you don't have a data plan that is, uh, going in every country. So it makes sense to have something that works on your phone if you want to order at a restaurant, something like that. I think it's a particularly adapted use case, but now we're also, Uh, we released two weeks ago a model called Pocket ETS that not only is on device, but CPU only. So, uh, there are already mods for AAA video games, uh, where the NPCs can be powered through these voice models. And now you unlock a completely new kind of, uh, of applications because on device models allow to do very large scale personalized content, uh, that will be, uh, economically not realistic with an API. So again, these kind of things is, uh, You know, if you want to make meaningful progress in that dire…

AI assessment note: “on device models allow to do very large scale personalized content, uh, that will be, uh, economically not realistic with an API”

Q Okay. Any, uh, lessons learned so far? I mean, as, as you said, you guys come from a very deep background in, uh, in AI and AI research. It's a whole different thing to build a whole like data center facility. How have you gone about it? And, uh, what, what are some things that, that surprised you and any lessons so far?

A As most new experiences as a founder, I relied on the knowledge of others, and so I was lucky to have a very, a few seasoned HPC experts and a lot of cloud software experts as well to build that solution. For me personally, and it's one of the things I love about my position at Mistral is that I get to discover so many new things and so many new problems I hadn't thought possible. Having to learn to like all of the different parts of building a data center, all of the different trades that you have to coordinate, uh, all of the potential, um, synchronization, uh, between all of the different trades. I mean, it's a huge building. It involves hundreds of people working on it. You have this, then when you, uh, stand up the thing, uh, you have to question what works. You have to filter through the blades that are faulty. It's just an entire new area of work where I get to see, um, Experts in their field go through things and try to explain to me what their daily work is. It's always fascinating to see, um, an expert in this field, like do something that you don't know how to do. I think the logistics of it, uh, and the timelines are also quite different from what I'm usually, um, dealing with in software and research. For new capacity to, uh, be built, you have to plan around, uh, Having energy available, you have to plan for the, uh, space to be available and on time. And so it's …

AI assessment note: “I think the logistics of it, uh, and the timelines are also quite different”

Q And who does the, all of this you have evolved towards an FDE model?

A So we have indeed a large, uh, FDE section. It's, it's a mix of software and And, uh, FDs, and we split our FDs into what we called, um, AI engineers and applied scientists. Um, and so, uh, applied scientists will tend to use the tools that we've just, uh, uh, talked about. So fine-tuning, continued pre-training, and the likes, where AI engineers will focus more on adaptation to the enterprise environment and figuring out what workflows to automate and all of this. They work with the customers to make sure that the use cases are indeed providing values and going to production, but it's also a fantastic way for us to understand what matters in an enterprise context and be faster at building the right platform.

AI assessment note: “It's, it's a mix of software and And, uh, FDs”

Q As I'm sure you've seen, there was, uh, for the last few weeks in startup and venture circles, there's been this whole, um, idea of the context graph as an infrastructure that made the rounds. Is that something that you think about or a layer that would, uh, basically, uh, enable one to know how the agents made a decision and, uh, how those decisions relate to one another?

A I've seen this indeed, and I think there are Two, uh, levels to that discussion. The part that you mentioned at the end where, uh, it's interesting to know how an agent came to. So in that discussion, when, when we talk about understanding how an agent came to a decision or an action, the game is really to understand how a human, uh, agent really made this decision. It's understanding how an enterprise does what it does. And it's certainly interesting. Uh, what keeps me up at night and what I really want to solve first is just the basic idea of gathering a workable enterprise context. Right now, uh, with, uh, any model, uh, and with a lot of effort, you will be able to get some connections to tools and you will ask your questions and your agent will do a bunch of things. It will realize that, oh, by doing Five API calls and three joins. I can probably get, uh, what Timothy asked immediately. What should happen is that, um, all of that, uh, discovery and all of that intelligence should be stored somewhere to be reused. It's not really how things happen. It's just basic knowledge about what the infrastructure of the company is. So knowing where the tables are, what they contain, how they're joined. So all of this, um, is, Compute that should be amortized, basically. And to me, it's really the entire game with the context engine, as we call it internally, is to be in a setup where…

AI assessment note: “I've seen this indeed, and I think there are Two, uh, levels to that discussion.”

Q Where does the edge fit in all of this?

A There are a few reasons to go edge. First, there are some regions where it's more convenient to be able to work without internet, and there are also a lot of capabilities that don't necessarily require a huge model. So if you just need something that goes voice to action, On any device, uh, today with, uh, typically the voxel models that we develop, this is doable. Again, an area where The more focused your use case is, the smaller you can make the model through fine-tuning or through just distillation in an even smaller architecture. I think voice to action is going to be a big use case. I think it will simplify a lot the current stacks for these types of things. There is also some privacy things where you could imagine all of the context consolidation stays On your personal device. And for most things, uh, you can deal with a small model, uh, that answers a lot of your questions. And then you potentially can get, uh, what goes out to, uh, another like cloud-based models. I myself take the train a lot. Uh, I like having coding assistance, uh, having, uh, DevStroll run on my laptop while I could on the train is, uh, comfortable despite the bad wifi.

AI assessment note: “There are a few reasons to go edge. First, there are some regions”

Q Another thing that you, uh, released reasonably recently, I believe is, uh, OCR three. What does that do? That enables you to just like, uh, scan Any, uh, any form, any document?

A Yeah. OCR is a huge use case in enterprise. Uh, a lot of our customers have, I mean, the typical example is KYC where someone will submit a form and you need to input that information in a structured way in your systems, or you need to reason about it. And so OCR, interestingly, is, uh, it, it's not the types of systems that I would have expected, uh, LLMs to really, uh, make large strides on. The visual reasoning and the visual understanding has gotten so good that it's, it's just an easier way to process things. Uh, in my mind, you have any sort of inputs, um, and you can get the, the data that you care about. As I mentioned, when you build agents, you have a different type of inputs for the tasks that you're trying to solve. Documents and visual informations are just a very, very frequent kind of kind of input. Uh, sometimes it's a lot cheaper Uh, to use a small OCR model to just get the text that you care about and then potentially post-process it or deal with it with another system than to run it through a large, uh, multimodal model that will, uh, basically do the same thing but at a higher cost.

AI assessment note: “Yeah. OCR is a huge use case in enterprise.”

Q So let's go into, uh, the enterprise reality of all of this. Um, so if I'm an enterprise or if I'm a sovereign and I want to deploy a Mr. Open source model, what is it that I do these days with everything that you, that you've built?

A The way we work with, um, enterprise, I mean, as you mentioned, like we have a few of our models that are open source and Apache and All of our clients are welcome to use them, uh, as they need. What we have seen in terms of success is that given the current stack, it still requires, um, a lot of expertise, uh, to manage to come to, um, actual value and, um, and things that go to production, basically. The way we interact is that we usually stand up our, um, Mistral AI Studio, which is our platform, and we can deploy all of our stack on the client's choice of deployment methods, so it can be on-prem, it can be on their VPC, it can be on in several places. The reason we do this is that it lets clients build where their data is, and without having to shuffle things around, which, as I've learned as a CTO, is something that you don't want to do ever. Because it asks, it raises a lot of questions, and it's quite a stressful thing to do. So once this is deployed, we then work with the business units to understand where their pain points are. Sometimes it's knowledge management, and I think it's the most well-known use case from the output, from the outside of the enterprise world. But it's also around automating core workflows for the enterprise. It's, you know, some Tooling that you wouldn't expect where one thing that we've done is around code modernization, uh, where you're, you …

AI assessment note: “we can deploy all of our stack on the client's choice of deployment methods”

Q What is the current constraint, uh, that, um, you guys are facing to make, uh, Mistral four when it eventually comes out, uh, do much, much better than Mistral three? Is that a question of Mistral compute or is that a question of, of data? And, uh, in particular, are you guys doing anything around synthetic data that you can talk about?

A Definitely compute and, uh, the current deployment that we have will help, uh, as it's going to be giving us a lot more Grace Blackwell capacity than we had in the past. And so that's, uh, something that we're very excited about. And when you add a compute, you also have to add data. And so we've been hard at work, uh, making sure that our, uh, data mixtures are, uh, as high quality as ever and growing in size. But as you mentioned, One of the ways to do this is through synthetic, synthetic data. In terms of, um, where we use synthetic data the most, I think a lot of the interesting work that's happening is for the post-training part where we can, um, build environments, uh, that look similar to, uh, an enterprise and then, uh, try to, uh, synthetically create queries that are hard and that will require multiple hops. And so all of this work, um, is in addition to the coding work, the reasoning work is really what makes the final model able to perform, uh, in the various, uh, environments that we work in. So before it was about, uh, accruing world knowledge and the, uh, web helps a lot with this. Now it's more and more about acquiring know-how. Uh, and for this, uh, it's really about Um, trying to find what our customers are trying to do, trying to replicate it inside of our training environment, and, ah, let the, the model run, basically.

AI assessment note: “Definitely compute and, uh, the current deployment that we have will help”

Q Fascinating. Just to unpack some of this, you can understand the reasoning steps that led to the explanation. You mentioned, um, in some of your writing label, uh, process reward models, PRMs, and the fact that this is not successful Yet. Can you unpack that, that part?

A So there's an outcome reward and a process reward. And the outcome reward is mainly like, uh, is the final answer correct or not? But then there's the whole explanation of the reasoning model, whether it leads to the correct answer. And so there's also research like, Hey, why would, why should we throw out everything the model generates and only look at the final answer? Can we get something useful out of this intermediate explanation? And the intermediate explanation is useful for several reasons. I mean, one, Is it has been shown like that this helps the model to generate the correct answer, whether the explanation is correct or not, but is a different, uh, aspect, but just the fact that it generates these intermediate steps is correlated with a more accurate answer. Then the hypothesis is if we can improve that explanation, maybe it gives even a better answer. Like even if, uh, maybe it even drives the accuracy higher. If you want to, um, like learn something, it's not enough to just see the final answer. You want to see the steps that lead to the Process reward models. They are also focused on training the model to, to reward the models based on that explanation. And so my statement that it is not so promising or useful was mainly based on the R one paper where they had a final paragraph at the bottom. I mean, this is already a year old, but they had a paragraph at the bott…

AI assessment note: “they found it wasn't worthwhile because of reward hacking.”

Q Great. What do you make of world models?

A Yeah. World models are, uh, also an interesting hot topic. So there is the whole world model aspect for more like images and physics and that stuff. So world models are basically Models that have like an internal model of the world. So they kind of simulate something internally, what you have externally. Like for example, if you have like a chess playing model, it has like an internal chess simulator built inside. So it can kind of make better predictions or predict the next states. I think that's a particularly interesting for robotics, but coming also back to LLMs. There was also a paper by Meta. It looked very promising to me as refinement or a next step for Code-based LLMs. LLMs for coding are still next token predictors. But in addition to that, what they did is they also try to predict the internal states of the variable. It's like there was like a during training and objective to, if you have a Python code to say, okay, this, at this iteration, when I, if someone would step through the code, this variable would have that and that value. And so this is in a sense, giving the model more context, more information about the training data. And it forces the model also to kind of like In quotation marks, understand training data better. So it's like, instead of just, you know, brute force, just what is the most likely next token? It has kind of like an understanding of what it…

AI assessment note: “It looked very promising to me as refinement or a next step for Code-based LLMs.”

Q If I understand this whole discussion correctly, all of this is about pre-training, right? And whether we can train a bigger model with, uh, you know, more data and more compute, but in conversations on this pod, a lot of the conversations have been about, uh, the importance of post-training and, uh, the, you know, building AI systems with pre-training plus, Plus RL. Where does that fit?

A That's a great question, and I think another piece that, that we, we didn't, you know, I don't think either of our blogs particularly hit on. One way I like to think about it is that pre-training is like the general strength training that you do in the gym. You go lift heavy weights, you improve your strength, improve your, your, your general ability, and then post-training is like the specific drills that you run to, um, to, to get a good, uh, at a particular task. So historically, the vast amount of compute has gone to pre-training. So just Building models that are more generally capable of doing many things, have a lot of knowledge, get to a point where maybe they, they have more knowledge than, than your average person. You know, I, I certainly don't know as much as, as ChatGPT for, for instance. And then the post training is both, how do you make it helpful? So, uh, you know, ChatGPT, you ask it to do something and then it actually listens to you, um, and, and tries to, tries its best to do it. But I think the other thing that, that we've started to see increasingly in post training is that you can start to post train Specific skills. So, uh, the model that's really good at helping you code uses a lot of the knowledge that you got from pre-training, but it's actually adapted to be particularly good for coding or, um, or the, or the, the model that's really good for legal w…

AI assessment note: “pre-training is like the general strength training... post-training is like the specific drills”

Q And then on your end, what have you learned or observed in terms of agents? What works? What currently doesn't work, but will work soon? How to manage them?

A I think there, there, there's two broad, um, things I, I, I've noticed for agents. So the first is, um, making the agents effective, uh, ends up being a lot like managing, um, junior folks on your team or, or at a company. So for example, the new intern who, who shows up on your team Um, you're not gonna go to the intern and say, hey, go fix our revenue for the year, double our revenue for the year, or something like that. Like, maybe you'll try that once, but you're, you're unlikely to, to see, um, the, the payoff from that. Um, instead of what you often do with junior folks is you say, hey, here's a first little task that you can do that in, to get to know this complicated code base. And here are the things that you might run into, um, because you, you've kind of done it before. When you give the agents that context, give them that ability to look at those things, then they can usually figure things out. The other bit is that when you have a new person on your team, you maybe won't give them access to all the production credentials and all the production database and all those things, but you're going to give them enough tools to, to be productive. Um, so sometimes there, there's this tension between, oh, I don't want my agent to go delete my everything in production. So I'm just going to Have it be hamstrung and watch every little thing it does. Whereas if you did that with …

AI assessment note: “making the agents effective, uh, ends up being a lot like managing, um, junior folks”

Q Let's talk about some of your work in alignment. You mentioned scalable oversight a bit earlier. What does scalable oversight mean?

A Scalable oversight is generally the field of alignment, which deals with using models to assist us in Aligning or other models. So in particular with grading the outputs of other models and checking them for safety or for correctness. It's, it's an interesting field. It's very relevant to both alignment and to capabilities. So generally in machine learning, in RL in particular, we have this problem of supervision. The whole RL relies on our being able to tell which Samples from the model are good versus which are bad. Math with a numerical answer, you can just check the answer, or in competitive coding, you can just check whether the code is passing the tests, and that's why we have seen a lot of progress in those domains. But in creative writing, for example, it's very hard to programmatically tell if one sample is better than the other, and historically people have used this RLHF framework, reinforcement learning from human feedback, but also we now we want to use Models to be able to grade responses of other models to provide critiques or feedback. And then there is a question of how do you use that feedback? How do you learn from the feedback? But yeah, the scalable oversight kind of deals with all of those questions. So using models to, to critique, to provide feedback, to supervise other models.

AI assessment note: “using models to assist us in Aligning or other models”

Q Do you want to maybe define mechanistic interpretability?

A Yeah. Mechanistic interpretability generally, uh, tries to at a low level understand what is happening Inside the model. So they are trying to find these things called circuits that are, you know, some parts of the model that you can isolate and understand and kind of model in your brain that correspond to certain behaviors in the models. And there over the last maybe three years, I think there has been some pretty major progress there. So we are still pretty far from the dream that we will Fully understand everything that happens in the model, but these tools are becoming increasingly more useful internally, uh, at Anthropic in particular, and also there is constant progress, and, uh, it's pretty fascinating work, actually.

AI assessment note: “tries to at a low level understand what is happening Inside the model”

Q Why is it so hard to truly understand what a deep learning model of, of any kind actually does?

A Deep learning models are huge. They have billions, trillions of parameters, uh, and They are doing some messy mathematical computation. You can understand what they're doing at some level. It's like a bunch of matrix multiplications and some, you know, rearrangement of vectors, but that's not a sufficient level of understanding. We want to understand it at a lower level, and it is very possible that that's just not fully possible. Like, it is some computational process that leads to some results. It doesn't have to be the case that you can Kind of describe it in human terms, uh, and kind of understand it very discreetly. I think also, uh, something that contributes to this complexity is just how many things the models are capable of doing. And they are not trained on some small isolated behavior in some small context. They are, you know, they know all of the internet. So all of the information in all languages is somehow encoded somewhere in the weeds. And then they also have all of these behaviors, all of these correlations between the knowledge. All of that is somewhere in the model and just like making sense of all of that is extremely hard.

AI assessment note: “Deep learning models are huge. They have billions, trillions of parameters”

Q question in, uh, all the current reasoning progress. Do we understand what does what and was it, what is responsible for what kind of progress? So if you take, uh, test time compute If you take, uh, the ability to, to search, uh, if you take RL, uh, do, do we know, uh, which one of those techniques, uh, we should turn the knob on, uh, to get better results?

A All of those techniques, they don't exist independently, right? Uh, RL is mainly kind of used to teach the model to use test time compute. Uh, so you first need to prime the model to, to set it up so that it outputs a bunch of tokens before outputting to answer. But then you, you spend the compute in RL so that it learns to output the right tokens. So in my mind, those two are almost kind of indistinguishable, the RL and the test time compute. RL is a method for training and test time compute is maybe just a more general concept. Yeah, you can potentially get to models that use test time compute without RL, but that's not how we are training them right now. Yeah. So I think the trend has been in spending more and more compute on the RL. And getting the models to make better and better use of test time compute. And the tools are also, of course, extremely important, like the web search that you mentioned, and also just the models being able to write Python code and run them, produce artifacts for you. That is extremely important for the product and for making the models useful to people. Conceptually, I think that's a little bit secondary. Like in my mind, the main thing is, you know, the, the RL and getting the models to, to think for a long time.

AI assessment note: “All of those techniques, they don't exist independently, right?”

Q What's, uh, working so far. So, you know, you, you hear talking to people, um, you know, some people talking about like running agents for like a couple of hours, uh, but then some people are talking about like agents running for like 24 hours, uh, 32 hours. Where are we in that arc? And, um, What is working? What is not yet working?

A There is this famous, uh, meter, uh, plot, which shows how long of a task AI is capable of, um, robustly automating, and it's been kind of consistently doubling at that time, uh, every half a year, I think, uh, and it's now in like some hours, uh, so maybe a couple hours. In terms of the methods that are working well, I think, yeah, right now it would involve some kind of a harness, uh, with a bunch of Agents that interact, uh, or, uh, that sequentially solve the task, and there has to be some kind of orchestration, uh, or maybe like some initial task decomposition, and yeah, and it's all not very well established, I'd say. It's, it's a new domain, and, uh, I think we are still figuring out how to best do it.

AI assessment note: “it's now in like some hours, uh, so maybe a couple hours.”

Q I know that you spend some time thinking about, um, the impact of AI on science and math. Um, same idea. Any predictions, uh, there? Like, do you expect, uh, important new discoveries to be made by AI solely by AI?

A Uh, it's a great question. And, uh, I think it's, In sciences, I think that's maybe a little bit more likely. Uh, although I, I also don't know very much about, you know, the life sciences. It feels that there, some discoveries can be made by potentially combining results from different parts of the literature and like proposing some ideas that turn out to, to be true. I think it's hard to imagine the AI making a discovery independently in like Add a main where you need experiments, because my understanding of a lot of science is it's about doing the experiments and you, you need some reasoning to guide what experiments you do, but you also need a lot of iteration and a lot of, uh, like actual, you know, things happening in the physical world. Uh, and at least for now, the AIs are not capable of doing that. In the mathematics, um, I think we will see the models getting better on proving technical Um, results, technical lemmas, uh, maybe including formalization and, uh, like things like lean, uh, the formal theory, improving language. I think the models, it's easy to imagine the models becoming better than humans at proving this technical lemmas, uh, quickly. I think the impact on mathematics is very interesting. Um, so it is improving the output of humans. Already, but it also introduces some noise, right? It also, like some of those rules will be incorrect, and they will be in…

AI assessment note: “hard to imagine the AI making a discovery independently”

Q I'm always curious, as an AI researcher who's like so deep into the very heart of all of this, if you zoom out, are you, are you still surprised by where we are? Like, from your perspective, are we well ahead of where you thought we would be a few years ago? Are we on track? Are we behind, possibly?

A I think it's easy to say we're on track, in hindsight. I think, if I'm being honest with myself, I think we're ahead of where I thought we could go. Um, starting work on LLMs in, in 2019 or 20 20, it's, it's kind of hard to believe, uh, the scale of everything we're doing, but also just what the models are capable of, of doing today. If you just, if you kind of looked at scaling laws back then, they were definitely pointing, uh, towards that direction. And, uh, some people really believe those deeply. I, I'm not sure if I would have bet a lot on, on that actually materializing and, and being where we are today. So, One interesting question that follows from this is where, where does that take us? If we assume the same, or if we assume the same kind of progress we've seen in the last five years, I think, yeah, this is going to be very, very cool what's going to happen in the next few years as well.

AI assessment note: “I think, if I'm being honest with myself, I think we're ahead of where”

Q You use the word, um, research taste, uh, which I think is, is super interesting. What, what, what does that mean? How would you Define that, and how important is that for a researcher?

A Yeah, it's, it's very important these days, and it's quite hard to quantify, but the few things that matter is, the first one maybe is your research is not standalone. This is what I was mentioning before, but your research has to play well with everyone else's research and has to integrate, right? So let's say I have some improvement on the model, but it makes the model five percent harder to use for everyone else. This is probably not a good trade off, right? Because you're going to slow down everyone else and then their, and their research, which would then summatively slow down the durable research progress. That's the first thing. Um, the second thing is being allergic to complexity. Um, but complexity is quite subjective is in terms of what people are familiar, but still this, we have those certain, I think, budget of complexity we can use in a certain amount of like almost research risk we can accumulate before things go bad. And so being aware of that and managing that is very important. So oftentimes we don't necessarily want to use the best performance version of a research idea, but we'd rather trade off some of the performance for a slightly lower complexity version because we think that will allow us to do more and more progress in the future. So these are kind of the main two things I think around research taste.

AI assessment note: “these are kind of the main two things I think around research taste.”