The Exchanges

Q Oh, is that why you guys stopped showing reasoning?

A That is part of it. So there's two reasons. One is to think about distillation, but the second, in some ways more important, is that we had this insight when we first developed the reasoning paradigm that it gives us a interpretability mechanism we had not been anticipating, because you can really read the model's thoughts. You can see exactly how it got to an answer, so you can interpret How, like, what was actually motivating that answer? Now, the problem is, if you train the model to have a chain of thought that looks good, then you lose all the faithfulness, right? It's just going to be like, the model knows that part of the answer that is desired is for the chain of thought to look a certain way, and so it may not be representative of how it actually arrived at that answer anymore. And so we were, we made an early decision to say we want to avoid any temptation to train these chain of thoughts to look at Favorable to look like something you could present to a user, and so that really made us lean out for multiple reasons, for competitive reasons, for safety reasons, from the idea of showing these intermediate thoughts.

AI assessment note: “That is part of it. So there's two reasons.”

Q At what point did you realize that like this nonprofit thing just wasn't going to work?

A In 2017, we started to think very hard about, first of all, how do we really achieve the mission? How do we actually build an AGI? What will that look like? And we started to do the math on compute, and you start to realize that it's gonna take a big computer, and we came across a company called Cerebris, which was building a unique piece of computing hardware, and the kind of computer that they were promising, we realized was going to be far advanced of where our compute calculations looked. As you start to realize if we could buy a lot of those computers, we can actually probably succeed at building an AGI. If we could get exclusive access to Cerebris, that could give us an overwhelming advantage. If we could buy very large data centers, that could be something unique as well. And the thing about nonprofit fundraising is I think that there is essentially a cap to what is possible there. And so Elon, Sam, Ilya, and I all agreed. That the only path forward for OpenAI, the only path to achieve the mission, was to create a for-profit entity associated with OpenAI of some form. And so we were committed to that direction, and that is something that we knew was the only way to achieve the mission.

AI assessment note: “In 2017, we started to think very hard about”

Q Is there a difference between reasoning and predicting? You mentioned sort of like predicting the next character, predicting the next word versus actually reasoning in first principles.

A I think they are connected in a deep way. So On the one hand, just predicting what comes next sounds like a pedestrian task, but if you really can predict the next word out of Einstein's mouth, you are at least as smart as Einstein. And you can make arguments, oh, well, like, you know, it's, but I, I think that those arguments fall flat, that there's something, there's something false there, because the point of prediction is not about being able to predict what is known. The point is you put yourself in a new situation you've never seen before and predict what comes next. And I think that there's something deeply connected to intelligence and prediction that there's a long story of academic literature and how you think about this compression. They're all kind of part of the same thing. Now, these reasoning models, the thing that I think is very interesting is that we train them with reinforcement learning. And so there's really back to the original open AI plan. There's two steps to it. The first is unsupervised learning. You train a model just by having it predict what comes next and And there it's much more static data. It's much more observational. Again, it's data. It's never seen before situations never seen before, but it is a situation that has already happened. Then you do reinforcement learning, which is you basically have the AI learn on its own data, right? You have…

AI assessment note: “I think they are connected in a deep way.”

Q If I were to say, like, open a Google document, write out sort of what you learned about yourself on one page from this whole starting to Sam getting out of state, to you quitting, to inspiring all this loyalty to the time off and then coming back, what would you write?

A I think I've just learned to just keep going for something that's worth it, right? If you have a mission that matters, then The fact of you keep going through the ups and the downs. There are going to be moments where it's It's all over. There's moments where it's, we're so back, and you just can't let those moments pull you off course. And I think that the degree of just personal resilience that you have to grow during these times, because if you're leading, people look to you for that steadiness, for that support, for the direction that the whole thing will go. And I think that a lot of what I've tried to grow with is to really Be able to both understand the details, right, of what we're doing, what the implication will be of a choice, but also be decisive. I think that, that there have been moments where I think I've been very much approaching OpenAI through a lens of uncertainty, of feeling like, I don't know what the right answer is, I don't know what the right way to build this technology is, or how do you answer these very thorny questions, but there's lots of people here who are very smart, who have very strong opinions, And so really try to understand all those opinions and figure out how to put them together. And sometimes that's the right thing. And sometimes that you realize that the opinions are mutually contradictory. They can't all be true at once. And sometimes …

AI assessment note: “I think I've just learned to just keep going for something that's worth it”

Q Is it coming up with novel ideas that you wouldn't have thought of?

A I'd say that where we are is we're getting close. So we've seen, for example, in chip design. So in the design of our own chip last year, we applied our technology to Trying to get a better fit to actually shrink the area used by the circuits, and there we found that the optimizations that the model produced were actually on our list, so it didn't come up with something novel and new that no human ever would have, but it implemented it faster in a way that we wouldn't have had time to accomplish. If you look at math and physics, we now are solving open math problems. We're solving Open physics problems and actually have resolved this particular physics problem recently in quantum physics in the opposite way that the community expected. And with a beautiful, elegant formula, it's like, it's really happening. So new ideas from these models, extremely doable. We're starting to see it in some of these domains now applying it in harder and harder domains or ones that require more real world context and things like that. We're starting to see it. We have a line of sight for how to accomplish it, but we have a lot of work to do.

AI assessment note: “I'd say that where we are is we're getting close.”

Q When I told people I was doing this interview, one of the common reactions is that they're fearing for their job and their uncertainty. What would you tell them?

A Well, I do think that this technology, it is uncertain exactly how it will play out. I think it is surprising how it will play out as well. Like the AIs that we have right now, the world that we have right now is not really something that was anticipated by science fiction. It's just different. And some inevitable conclusions, I think, actually turn out to not Don't quite look the same way when they come to pass. So I believe it's always easiest to see what you lose, right? And the change is coming. There's no denying that. That is absolutely the case. But it's much harder to see a priori what you gain. And as an example, just think about Uber being described as someone in 1950. You have to think about computers. You have to think about mobile phones. You have to think about GPS. And it's all so that you can get a car to a Appear where you, where you are in three minutes. And like, that's actually crazy. If you think about that level of technological investment for that kind of use case, but it really happened. And it didn't just happen for that one use case. It happened for thousands, for tens of thousands, for millions of other use cases. And so I think that my view of AI is it is about empowerment. It is about human agency. And that that does mean that some of these institutions, jobs, these kinds of things, that there will be things that we thought we could rely on that tur…

AI assessment note: “the question to lean into is what do you gain and how do you benefit”

Q Are we near the point where AI makes AI go parabolic?

A I would say we are in this phase where you apply AI to its own development process and it's going to go faster and faster. And that is something that's been happening really. I mean, certainly since ChatGPT in many ways, right? We use ChatGPT to make our development process. 10%, 20% faster. Now we have these amazing coding tools, which have truly revolutionized how Software engineering is done, and most of what we do in the production of models is bottlenecked by software. It's about implementing these systems. It's about scaling them up. It's about managing these massive computers. And we're going to be hitting a phase soon where the AI will also come up with its own research ideas and test those out, run experiments. And so I think that the speed of iteration and innovation is going to continue to increase as a result of what we're producing.

AI assessment note: “we are in this phase where you apply AI to its own development process”

Q But if we're computer constraint, like, how do you decide who to serve? Like, why are you serving me when I'm, like, trying to make an image over, like, solving cancer?

A Well, this is going to be the most important question for society to answer. Where does the compute go? What problems are worthy? And there's lots of worthy problems, but you need to prioritize them, because you only have so much compute. And so one thing we really believe in is that everyone is going to need access to compute. And so that's why we have a free tier of ChatGPT. We've really Put effort into making sure that people are able to use this technology, that it's widely available, because we believe that is core to what we're doing here. We think that putting this technology in people's hands, that empowers them, that lets them achieve goals. It helps them also understand the technology, right? It's something that helps them then shape how does this technology slot in? You could take a very different approach and say, well, it's all about the ivory tower. It's all about the just solve the problem, and we will then distribute the Technology breakthroughs in some way. And I think there's merit to that as well, but that's not where I'd put the, the balance of, of what we do, right? I think that that is very much a, like, we do want to make great strides on specific problems, but I think that that should be in service again of the, we want the benefits of this technology to be broadly distributed.

AI assessment note: “one thing we really believe in is that everyone is going to need access”

Q Do you think the models evolved to tell us what we want to hear if they're based on reinforcement learning? So if I lean left, it's going to tell me an answer that leans left, or if I lean right, it's going to give me an answer that leans right?

A Well, so we've actually gone through an evolution of how we train the models to user preferences, and that we've seen that at one point, like last year, that the models really did start to lean into telling you what you wanted to hear, saying, oh, that's such a great answer. And we reacted to that. We said that this is not how we want our models to operate. And we made changes because the true thing we want the models to be aligned to is helping you solve your goals, your long-term goals, right? And maybe in the moment it feels good to be told that was a great question. Best question anyone's ever asked, but that's not what you actually want. Maybe there's some people, but it's not, it's not what most people truly want. And so we've actually made great technological improvements to make sure that our AI training does not result in what is called hacking the greater, right? That we really want to make sure that there is a good signal there that is about the goal, not just your short term, what's going to get you a quick hit. And that to me is maybe the most important Part of the vision for where our personal AI, personal AGI is going to take us is to really make sure it's not just about something that looks good in the moment. It's really about alignment with your long-term wellbeing, your long-term goals, the thing that you actually want. And that is what I think will most empo…

AI assessment note: “models really did start to lean into telling you what you wanted to hear”

Q Are other countries stealing advancements? I've been reading a lot about distillation.

A There's certainly a lot of attempts to distill models. And that comes from companies in the U S it comes from, uh, from all over the world. But I think that it misses the core point, which is that the way this technology is developing is it is on an exponential. And anytime we have a model, we've already moved on to the next one. We're already moving to the next level. So we put in a lot of effort to protect against distillation, make it harder to do, especially with things like chain of thought and other parts of the model that are not really necessary to get the benefits to someone, to get the outputs to someone, but that the core advantage that we have, the strength that we're building up over time is really about not just any one model. It's about the Machine that makes the models.

AI assessment note: “There's certainly a lot of attempts to distill models.”

Q Is there a consequence, do you think, for the United States not being the first country to reach AGI?

A Well, I do think that leading an AI is very critical for America, because I think that this is how you can ensure that democratic values are protected and preserved, and I think that every country is also starting to realize that they need some sort of sovereign AI strategy. They need to, if this is becoming the basis of economic security, of national security, They need to participate somehow. And if you look at a lot of the efforts by the United States to think about how to manage chip exports, how to think about technology exports, there's something where if you lean too far out, then everyone else has to develop their own competitor or rely on someone else who's, who's building this. If you lean too far in, then maybe you lose your advantage and The question is, how do you balance those? How do you maintain your leadership? But leadership is not just about being ahead. Leadership is about also bringing along the world with you.

AI assessment note: “leading an AI is very critical for America, because I think that this is how”

Q If one frontier model puts safety as a primary concern, and another frontier model doesn't, how do you view that competition playing out over time?

A Well, I think we have found that safety is actually a core product feature. Like no one wants a model that is not aligned with them, right? You want a model you can trust that does the right things in any circumstance you give it. And so we have invested, I think we've actually invested possibly far more than certainly people perceive and possibly more than any other lab in safety. Right? That we have in ChatGBT the broadest deployment of AI, these language models in the world used by the most people. We have to care. We've always cared, but you really see it in terms of us being able to bring this technology to so many people. So I don't think that there's a sustainable state where the people who are building this technology and having successful products are not also investing super hard in safety. And I think that actually the challenge Is a little bit about if you step back, because there are some aspects of what it means to deliver safety that are not necessarily short-term. You have to think long-term for not just your business, but for what it is that you're creating, and some of this is about how you train the model. Some of this is about how do you get your feedback loop, but I would just say that we are committed to safety as part of our mission, and that's something where I think it has played out In our products and in the world. One thing that people also miss is t…

AI assessment note: “I don't think that there's a sustainable state where the people who are building”

Q How far away do you think we are from that?

A Well, data centers in space has a lot of, has many technical problems associated with it. Even for example, the data centers we build today are very finicky, right? They're these massive machines with very breakable, very expensive components. We've had many issues in the past where the cables were just too taut, just literally like too, too tight of cables. And then you get signal integrity issues and the computer doesn't work. And so figuring out how do you maintain Systems. Today it's people go and physically pull them. Probably we'll move to robotics. So I think figuring out how to solve some of these technical problems are going to be very important dependencies as we think about putting them in, you know, people talk about putting data center in, you know, various difficult locations. Um, space feels like a, like a grand challenge, but I think that we are going to have such need for compute that we need to be thinking about all options.

AI assessment note: “Well, data centers in space has a lot of, has many technical problems”

Q That's the most positive sort of view of the future. What's the most negative one you can imagine?

A One thing that's very interesting about how technology has played out to date is that it's really been about contorting ourselves to the machine, right? You think about how many people work where you have this box and you're typing away at it and you're getting your carpal tunnel and your shoulders are hunched and all of those things that were not natural, right? That's not really what we're designed for. And we're going to be moving to this world where it's not just that you're Doing work with your computer, so your computer actually does work for you. And that is something that presents opportunities. I think it presents risks. I think we need to figure out how to mitigate those. Like, one core thing at the end of the day is that if you have machines that help people actualize their goals, right, that's out there doing what you want, sometimes people have conflicting goals. How do you resolve that? How do you decide what the bounds are on what an AI will help you with and what they won't? Really trying to figure out how does this slot into society? How do you make sure that the benefits don't just go to one corporation, one set of people, but that actually do lift up everyone. We need to raise the floor so that everyone has access to a great life, this technology, and are able to do things with it. And I think it'll correspondently also lift the ceiling. And so I think we're …

AI assessment note: “presents risks... We need to raise the floor so that everyone has access”