The Exchanges

Q So I'd love to do a little bit of a deep dive slash educational part on, um, the whole reasoning model aspect, uh, because, uh, as you just mentioned, since it's so new, uh, some people truly understand how those work. Many people Don't. At a very simplistic level, what is a reasoning model and how is that different from, um, your sort of base LLM?

A So a reasoning model is like your base LLM, but before giving you the answer, it, it thinks what people call in the chain of thought, meaning it generates some tokens, some texts that's meant not for you to read, but for the model to give you the better, the better answer. And while it does this these days, it is also allowed to use tools. So it can, for example, in its thinking, so-called thinking process, go and browse the web and to give you a better answer. So, so that's the superficial part of the thinking models. Now, the deep part is that you start treating this thinking process as part of the model, basically. So it's not something the model generates and it's an output for you. It's something you want to train, right? You want to tell the model you, you should think well, you should think so that the answer after this is good in, in whatever way. And this leads you to a very different way of training the model because models were in usually trained with just gradient descent, the way deep neural networks are trained, meaning you say, predict the next word and you do a gradient, you Differentiate your function from the model. They're not fully differentiable, but you approximate it, and you train your weights to, to do that. And that, it was quite amazing that doing just that, you could make a chat. But with the reasoning model, you can do that because there is this rea…

AI assessment note: “a reasoning model is like your base LLM, but before giving you the answer, it, it thinks”

Q time when talking to people, um, is that part of the reason why people are so optimistic is that there is a lot of low hanging fruit, very obvious things to improve, uh, for those models in the next few months. First of all, do you agree? And second, can you give us some examples, like obvious thing that you need to fix next and that the industry will fix?

A Yes, there is a ton of extremely obvious things to fix. Um, The larger part of this ton is, is just hard to talk about on a podcast because it is in the engineering part. You know, every lab has their own infra and their own bugs in the code. Machine learning is beautifully forgiving in some sense, in contrast to old software engineering, which would just yell at you when you made a mistake. You know, our Python coded will generally probably run except much slower and give you worse results if you run it wrong. So you realize, oh no, it was wrong. And You improve it and the results get better. These are huge distributed computing systems. They're very complex to run. So, so there is a huge amount to improve and fix and understand in the process about just how to train your model and how to do RL because RL is more finicky than pre-training. It's harder to do. Right. So every day, this is our day to day work. On top of that, there is data. You know, we used to train on just like Common Crawl, basically. It's a big repository of the internet that people just scraped without regard of what. And some things came in, some didn't. It was a mess. So now, of course, every larger company has a team that tries to filter this and improve the quality. Um, it's a, but it's a lot of work to, to really extract better data. Now synthetic data is becoming a thing, but when you generate syntheti…

AI assessment note: “Yes, there is a ton of extremely obvious things to fix.”

Q impressive in particular as a user. I think that, uh, the, uh, 5.1, uh, moniker doesn't do justice, uh, to the evolution between 5.1 and five. It feels like a, a, a much larger improvement than the number would, would indicate from, uh, again, my perspective as a, as a user. Walk us, uh, maybe through the evolution of, uh, from GPT-IV to five to 5.1, what has actually changed?

A That's a very tough question. I think less than you think. It's, um, no, I mean, the, from GPT-IV to five, I think the biggest thing that changed is reasoning, meaning RL and synthetic data. As I told you, the pre-training part in that timeframe was mostly about making things cheaper. Not making things better. So of course the price has changed dramatically too, right? Thousand times I think are some of these orders of magnitude. The main improvements from four to five is adding reasoning with reinforcement learning and this allowed to generate synthetic data, which also improves the model. So that's the, that's the big picture. In addition to that, ChatGPT is now a product used by a lot of people. So The, the post-training team has learned a tremendous number of lessons, and it's added, you know, things clearly experimented, wanted the model to be very nice to you, then turned out to be too nice. Then now when a lot of people use it, you need to be really careful about safety, right? There may be people that are in distress using the model. The model needs to do something reasonable in these cases. It was not trained for it before. Now it is, and, and it makes the model much better But you know, in the same time, you don't want to refuse to answer any question that has any sign of anything. So, so as you work on these things, you make the model much better in use, not, not jus…

AI assessment note: “from GPT-IV to five, I think the biggest thing that changed is reasoning”

Q a frontier agent decoding model, a trend on real world software engineering tasks designed for long running workflows and using compaction to operate across multiple context windows in millions of, of, of, of tokens. So I'd be interested in unpacking some of this. What does that mean to run for a very long time? Is that an engineering problem or a model problem? And then maybe a word on compaction.

A So it is an, both an engineering and model problem. Um, you know, you want to do some engineering task, like write a, you have some machine learning idea. You want codecs to implement it for you, test it on some simple thing, find the bugs. So it needs to run this thing. This is not something you would do in an hour, right? That's It's something you'd spend a week on. So the model needs to spend a considerable amount of time because it needs to run things, wait for the results, then fix them. The model is not like it's going to come up with the correct code out from thin air, right? It's just like us. It needs to go through the process. And oftentimes in the process, since it was not trained on anything very long in its, in its training, or maybe very few, but, but nothing, certainly nothing that went on for a week, it can get lost. It, it can start Doing loops or doing something weird. That's of course not something you want. So, so, so we try to train in a way that makes it not happen, but, but it does. So, so, you know, how can you make the model actually run a process that requires this larger feedback loop without tripping up? And the other thing is transformers have this thing called context. So they, they remember all the things that they have seen in the current run. And that can just exceed the memory available for your run, and the attention matrices are n by n, where…

AI assessment note: “So it is an, both an engineering and model problem.”

Q at the end of a huge week or couple of weeks with release of GPT-Five. One, GPT-Five. One codex max, GPT-Five-One Pro, as well as Gemini Banana Pro, um, GROC-Four.One, almost three. So this feels like a major valuation of, of that narrative. What is it that people in Frontier AI Labs know about AI progress that at least parts of the rest of the world seem to not understand?

A I think there is, there is a lot to unpack. So, so, so I, I want to go a little slower. There is this thing that's happening in AI, and in AI, every week now, a lot is happening. You know, new model, coding, doing slides, self-driving cars, images, videos, you know, there, there's, it's, it's a, it's a nice field that doesn't make you be bored for a long time. Um, but through all of this, it's sometimes hard to see the fundamental Things that are happening, and fundamentally, if you look at AI progress, it's been a very smooth exponential increase in capabilities. This, this is the overarching trend, and there has never been Much to make me at least, and I think my, my colleagues in the labs believe that this trend is not happening. It's a little bit like Moore's law, right? We, we, Moore's law happened through decades and decades, and arguably you would say it's still very much going on, if not speeding up with the GPUs, but of course it did not happen as like one technology was bringing you there for four years. There were, it was one Technology, and then another, and another, and another, and another. And, and this went on for decades, right? So, so from the outside, you see a smooth trend, but from the inside, of course, You know, progress is made through new developments in addition to the increase of computer power and better engineering, and so all of these things come t…

AI assessment note: “fundamentally, if you look at AI progress, it's been a very smooth exponential increase”

Q And that started in high school as a kid?

A Yes, I was definitely very into math in high school and into computers also later in high school. Yes, I did my studies in Poland. I went for a PhD in Germany. It was theoretical computer science and mathematics PhD. So I very much I'm a mathematician. Yeah. I, I was always fascinated by, you know, how, how, how, how is this thinking going? What is intelligence? As, as a child, I always wanted to like emulate the brain. They thought, well, okay, maybe higher level explanations are more interesting. I did research in logic, but a little programming, but then there was this opportunity to join Google just as the deep learning was starting off. I already had my tenured position in France. And the French system has this beautiful thing that you can take a leave of 10 years.

AI assessment note: “Yes, I was definitely very into math in high school”

Q So there was a lot of pushback against the idea?

A Not against the idea. Google was also an amazing place at that time that they would very happily let you work on whatever you wanted. But, but I don't think there was widespread belief in doing multiple tasks with the same model, not to mention You know, this idea that you, I, I still find this idea that you take basically the same model as transformer. Like now there is a bunch of changes to it, but you could in principle take the same architecture as the decoder from the paper, train it on all of the internet, and it will basically start chatting with you. That would have back then definitely sounded as a worthy dream we maybe had as a dream, but not the reality that you expect five years later. It's very lucky that it actually works so well, right?

AI assessment note: “Not against the idea. Google was also an amazing place”

Q In between models or different versions of the models, are the other releases aligned with Pre-training efforts or sometimes you have like one big pre-training effort and like several models that come out based on that?

A There used to be a time, not that long ago, we have a year distant past, where, where, where the models were, did have an alignment with, with technical stuff, right? So they would align whether with our, either with RL runs or pre-training runs. That's why you had, um, A beautiful model called four O, which was aligned with a pre-training run, which was obviously worse than the O-three, aligned with an RL run that was the follow-up to O-one naturally, because you couldn't use the name O-two. But it was slightly better than the O-four Mini, because that one was Mini. And, you know, we had this beautiful model picker, and people kind of thought this was not the best naming for some whatever reason. So Um, no, I mean, it was fairly obvious that this was a very confusing, right? So, so, so, so now the naming is by capability, right? GPT-Five is a capable model. 5.1 is a more capable model. Mini is the smaller model that's slightly less capable, but faster and cheaper. And the thinking models are the ones that do more research, right? In that sense, the naming is detached from any technical, in particular, you know, 5.1, maybe Just a pre-training, sorry, post-training thing, but maybe 5.2 is the newly pre-trained model, or maybe not, but, but, but the naming has detached from the technology, which also gives some, you know, as OpenAI has grown, there is a number of projects, right?…

AI assessment note: “the naming has detached from the technology, which also gives some, you know”

Q Great. What is, um, next. For pre-training, we talked about data, we talked about engineering, a big GPU compute aspect to this. What happens to pre-training in the next year or two?

A Pre-training, as I said, I, I think it has reached this upper level of the S-curve in terms of science, but it can scale smoothly. Meaning if you put More compute. You will get better losses if you do things right, which is extremely hard, and that's valuable. Um, you don't get the same payoff as, as, as pushing HRL, but it generally just makes them all more capable. And that's certainly something you want to do. I think what people underestimate a little bit in, in, in the big narrative is, you know, OpenAI three, four years ago, I joined even before that, was a small research lab. With a product called API, but you know, it was not such a big, there was no GPU constraint on the product side, for example. All GPUs were just used for training. So it was very easy as a decision for the people to say, you know, we're going to train GPT-IV. This will be the smartest and largest model ever. And what do we care about small models? I mean, we care of them as to make, like, to debug the training of the big model, but that's it. So GPT-IV was the Smartest model, and it was great, right? But then it turned out, oh, there's this chat, and now we have a billion users, and, you know, people want to chat with it a lot every day, and you need GPUs. So you train the next, like, huge model, and it turns out you cannot satisfy this. Like, people will not want to pay you enough to chat with the …

AI assessment note: “reached this upper level of the S-curve in terms of science, but it can scale smoothly.”

Q What would it take for RL to generalize? Is that better evaluations? Like you, you guys released a GDB Val a few weeks or months ago, uh, to sort of measure performance against sort of broad economic sectors. Is that, is that Part of what the system needs?

A I think this is a small part of it. I, I, I think that's one part, but, but if you think of economic tasks, you know, making slides is important. They're following instructions, doing calculations. It's not math, but it's still very verifiable, right? What, what I'm thinking about is when you do pre-training, you take the internet and you just say, ask, what's the next word? You know, you could think Before you ask what's the next word, obviously. Now you don't want to think before every word, probably, but I don't know if you ever looked at the training data for like a real pre-training run, because I think people mostly don't realize how bad this is. Like hotels.com is a great website compared with the average chunk of 2000 words from the internet. It's, it's a mess, right? And also a miracle that from this, the pre-training process Gets you something reasonable. So you probably don't want, you know, imagine you have a hotel website telling, you know, it's a beautiful vacation. You don't necessarily want to have a very long chain of thought before that, right? If it was written by a person, there was probably some kind of thinking that went into it. Maybe, maybe not as elaborate as the math and coding thinking, but maybe there was something going on. So maybe you want a little bit of thought before at least some of the text and that our models Can't do very well yet. I think …

AI assessment note: “I think this is a small part of it.”

Q How are the research teams organized within OpenAI?

A Um, They're organized. They're not very organized. I mean, we do organize them, but some people have managers, and we sometimes talk to them. Um, no, but mostly people find, like, Projects. There, there, there are things to do, right? Like improve your multimodal models, improve your reasoning, improve your pre-training, improve whatever this part of the infrastructure. People work on that. You know, as we go through these parts, right? There is infrastructure, pre-training, reasoning. I think the parts are the same for, for most of the labs. So, so there will be teams doing these things. And, and then sometimes people, um, change teams. Sometimes new things emerge. There's always some smaller teams doing like more adventurous stuff like diffusion models at times. Then, you know, some of the more adventurous stuff like video models gets big and, and, and then maybe they need to grow.

AI assessment note: “mostly people find, like, Projects. There, there, there are things to do”

Q In 5.1, you have additional granularity in terms of telling the model how long it should think. By default, how does the model decide how long it should think?

A So the model sees the task, it will decide on its own a little bit how long it should think, but you can put, give it an additional, it's trained with an additional information that can tell it to think even harder, and then it will think longer. So you have now the ability to steer that. I, I still think it is important to realize So this is the fundamental change that came with reasoning models, that using more tokens to think increases your capability, and it increases it, given the computation, way faster than pre-training, right? So, so if you give GPT-V the ability to think for long, it can solve tasks that are, you know, we had these gold medal at Mathematical Olympiad and Computer Science Olympiad. So, so they're amazing abilities. At the same time, The, the fundamental training method of, of reasoning is very limited to science data. So, so it's not as broad as the pre-training, which I think like pre-training models felt kind of almost uniformly good or bad at things. I mean, this was still not uniform because it's not like teaching humans, right? But the reasoning models are even more, people call it jagged, right? They have amazing capabilities somewhere and then close by Not so much. And that can be very confusing. It's something I always love that. It's weird because you can say the model is amazing at Mathematical Olympiad. At the same time, I have a math book fo…

AI assessment note: “the model sees the task, it will decide on its own a little bit”