The Exchanges, every show

Q And isn't cooling technology, uh, that is being used as something that's, uh, well understood and it's just getting deployed, or is there like fundamental new things happening in cooling right now?

A I think liquid cooling has been around for some time, ah, but has never been deployed at this scale, and so the innovation is more around how to make it reliable, how to make it, ah, cheaper, ah, more scalable, right? So there's a lot of innovation around that. There's also a lot of new innovation, new kinds of liquids, new kinds of materials that can absorb heat better, ah, because anything that can improve the efficiency of heat transfer Uh, is very important for data centers. So we can then run the chips hotter, right? And there is a direct correlation between running a chip hotter and how powerful the compute is. So the hotter the chip, the more memory bandwidth you get, the more flops you get. And so there's more, there's a strong payoff. If you can cool well, that also means you can produce more intelligence.

AI assessment note: “innovation is more around how to make it reliable, how to make it cheaper”

Q addition to being the application business, consumer and enterprise, and then being in the model AI research, uh, business, and then the computing data center business, since that OpenAI is in the, Uh, chip business, if that's fair, sort of completely full stack, uh, but I'm, I'm curious, and we'll, we'll go into some details about, you know, later, later, but into, like, overall strategy, where, where does that fit?

A As we begin to, as a pretty big fraction of the world's population, AI usage is exploding, uh, inference is obviously becoming a big fraction of our work. It's consuming a lot of compute. And one of the other realizations is because we know what is the work toward exactly, what is the model we want to run, uh, we can co-design the hardware. To be super efficient in delivering those models, right? And so the, the strategic thesis we have in here is how do we take advantage of knowing what the end workload is, what the model itself is, and design chips, um, that are very efficient in serving those models. So it really allows us to drive efficiency advantage, drive more tokens per watt. So the key metric that Jalapeno is optimizing is maximizing the number of tokens you can produce per watt. And because the world is constrained by power today, so the more tokens you can produce for the same amount of watt water, it's better for everyone. So we look at it as a very critical ingredient in scaling how we deliver intelligence to the world.

AI assessment note: “look at it as a very critical ingredient in scaling how we deliver intelligence”

Q Including chips. You also released a few weeks ago, uh, MRC. Which is a networking protocol. Walk us through that. What is it and why is it a big deal?

A It's a new, uh, Networking, uh, protocol routing technology, if you will, uh, uh, uh, to scale these really large cluster fabrics. So imagine you have, uh, a 100,000 GPUs, uh, they need to be connected together, and when you're doing large training runs, uh, they're constantly communicating with each other because these models are so large that The processing of the models is happening over the entire 100,000 GPU plus, for example. And you can imagine, imagine the number of links and switches and NIC cards that need to be there to connect all of these chips to them. At this scale, stadiums are fun, right? It happens all the time. Uh, you can't really even enumerate all the ways things could fail. So the strategy beyond MRC is how do you design algorithms and protocols that can gracefully And mask all these failures, and make sure that the training workload does not get impacted, right? It just, the network is an abstracted system that the training job does not have to worry. It's always going to be there. It's always going to find a path, even if a link fails. So it is all about reliability. It is all about availability. So how do we design protocols that contain the complexity of such a, such a big cluster? And make sure that we don't get stopped because of failures, which are very common in the system. So MRC is like a multi-part spraying protocol, where you can spray packets…

AI assessment note: “It's a new, uh, Networking, uh, protocol routing technology, if you will”

Q uh, it's a bit of a new world, right? So obviously not quite a pivot because obviously the all AI research is, uh, going, uh, you know, full speed ahead, but like building a whole new business within the, the, the company, is that, is that fair? Is that how people think about it? Because at least building models is one thing, building data centers, it's a whole different world.

A Yeah, I mean, I think OpenAI has always had a fundamental belief that computers are the foundation of everything, right? Computers are the foundation for intelligence, and the way we keep continuing to scale intelligence and distribute intelligence is by having compute, and so that has never been different, and it's never always been the belief. I think what's becoming clear is to build the kind of compute we need and at this scale, Uh, we have to not just rely on getting compute from our partners. We increasingly have to take a much more active role in building and getting that compute that we need. So it does, uh, absolutely feel like a new muscle that we're building in the company.

AI assessment note: “So it does, uh, absolutely feel like a new muscle that we're building”

Q It's a closed loop. And by the way, you mentioned Texas and rural areas. Uh, since we talked about data centers at the beginning of this conversation, uh, why, uh, do, uh, OpenAI and other companies pick rural areas? Like, how do you select a site for a data center?

A Uh, so many factors. So one is, of course, land, like, plentiful land. Uh, number two would be, uh, permitting. Like, can we build these things? And we want to build these things Such that they are, uh, not affecting any neighbor roads, right? So land that is somewhat removed as their ideal candidate. Of course, access to power, right? So a strong grid, strong gas, uh, availability, all of those are important factors. And then four is neighbor, right? So how quickly can you build these things? So availability of labor, construction labor, qualified electricians, bloggers, All of this can play with. So all of those factors go into every single site selection decision. Um, and I know we obviously Texas isn't popular because it fits a lot of these criteria, but it's not the only state. I mean, we have data centers all around the market in LA plus.

AI assessment note: “so many factors. So one is, of course, land, like, plentiful land.”

Q uh, all of the sense in the world, uh, given the scarcity. Um, and then, uh, it seems that Target has evolved, so from, uh, the, what was going to be a joint venture with Oracle and, uh, SoftBank, Um, to what now seems like, it seems like it's more like an umbrella term for the complete strategy these days is a, is a fair way to describe it or?

A Yeah, yeah. I mean, we look at Stargate as our compute strategy, and it, uh, is varying degrees of, uh, us designing or building the compute for ourselves. Uh, for example, with, uh, with Oracle Close Partnership, uh, we help them, uh, design, we help them with how to operate AI compute, which is effectively a new kind of compute. Uh, for us, uh, we work with SoftBank Energy, which is public. Uh, we basically have co-designed the one shell with them, and they're executing on that one shells, and, uh, we will be kind of figuring out how to operate our chips, uh, in these data centers ourselves, using the new chips. So Stargate to us is that umbrella strategy for across all of these different things, and think of it as an evolution That will continuously be on, because it's never going to be tomorrow we wake up and do only one kind of way of building compute.

AI assessment note: “Stargate to us is that umbrella strategy for across all of these different things”

Q Okay. So therefore, uh, financing is outsourced to, uh, apartments. Okay. We talked about jalapeno, like, let's, let's go into a bit more detail there. Um, because in particular, it seems that, uh, you guys went incredibly quickly in designing it. Like I read somewhere in nine months from design to tap out. So maybe Maybe walk us through that, and what was the reason it went so quick?

A Yes, it was incredibly quick. Uh, nine months is very, very fast. Probably the fastest I've seen in my career. Uh, I think several reasons. One is, uh, it, it's a team. It's a strong team. Uh, they have, many, many of the team have designed TPU chips at Google in the past. So very well-experienced team. We have a great partner in Broadcom that has a very strong track record of delivering XPOs, ASICs. So I think a strong partnership with Broadcom and, and making this happen. Three, I think, ah, perhaps an OpenAI unique point, ah, in most chip companies, or when you design chips, you don't know what you're designing it for, because you're a vendor, the customer who eventually runs the workload is someone different. Ah, there's a unique advantage here of us knowing what the future models might look like, and therefore, being able to short circuit A lot of the decisions you need to make, design decisions you need to make on the chip side, so that, that's super helpful. And finally, increasingly, AI itself, helping design and optimize the chip, that is usually the one that takes the longest time, because human, you're basically limited by how many human, human, much human time is there to process all this data and run the experiments, and we can do a lot of those iterations much faster.

AI assessment note: “I think several reasons. One is, uh, it, it's a team.”

Q To start, some people describe what's currently happening in the world of compute data centers as the largest infrastructure build out in history, bigger than, uh, the highway, uh, bigger than, uh, the railroads. And, uh, I'm, I'm curious, uh, one, if you agree, and, and two, what it feels like on, uh, from the inside, like what, what's, uh, how do you view what you're currently building at OpenAI?

A It definitely feels like one of the largest things humanity has ever built, effectively. Um, ah, definitely bigger than many of the things that I've heard of. I'm not old enough to have experienced the highway build out, but, ah, no, it's, ah, feels exactly like what it's on. It's in the, in the belly of the beast, so to speak. Ah, every day is, ah, we are making, we are making decisions, ah, we are on compute, That, historically, from my previous role, for example, at Intel, we'd probably take months to make, given the magnitude of the, those decisions, but the demand is so insatiable that, and it is growing so rapidly, ah, that we have to move very quickly. So, it's, and it's an intense time, but it's probably the most exciting thing an engineer would want to be part of.

AI assessment note: “It definitely feels like one of the largest things humanity has ever built, effectively.”

Q so I think like everybody knows that data centers have been built, uh, but, you know, gun to one's head, like, I'm not sure that everybody could say, well, what is actually being built? Because we've been building data centers of the industry for cloud, uh, for, for decades at this point. So what is politically different and new about, uh, the data centers that we're building for AI today?

A I think the biggest probably, uh, is, is the scheme, right? So we are, Essentially building large supercomputers, as we think about AI, and, ah, as we build intelligence and deliver intelligence, and models become more capable, we use it for more, more and more complex tasks, we need more and more bigger computers, effectively. And so I think the way, the best way to visualize data centers is giant factories, right, that are turning, ah, electrons into tokens, ah, that's a popular phrase nowadays. But I, it's, it actually has a lot of, a ring of truth to it. So how do we take power, how do we take those electrons and actually use it to power chips that effectively are delivering intelligence? Uh, and the way I visualize it is large football fields, uh, liquid cooled because these chips run really hot. Uh, the temperatures on these chips are very, very high. And so you'll cool them with liquids. You can't cool them with air. So a lot of liquid cooled, uh, basically refrigerators effectively, uh, that are sitting in the, alongside the building.

AI assessment note: “we are, Essentially building large supercomputers, as we think about AI”

Q problems, um, ahead in the next, uh, three years. Very paranoid about the surprises ahead. Um, which sounds like a very healthy approach. Uh, so how do you, how do you think about, um, that? Is there any way to mitigate that or is just like how do we play, uh, do I can believe that this is the future and, you know, we should all just go, go, go.

A We have deep conviction in scaling, right? And, uh, history has borne us out. So effectively our Yetni, for example, has tracked. We tripled compute and we tripled revenue. And we believe that, I mean, that continues to be true, demand far outstrips. Compute supply today. So anything we can bring online, we consume immediately. So there's no compute that is going to waste for us. So I think that conviction has not changed whatsoever. And if anything, we are seeing that scaling laws on research and training continue to hold, and potentially the pace at which we are doing research is accelerated. Right, because of AI itself, so AI is doing a lot of AI research now, and so one of the subtle implications of that is, Is previously our researchers used to run experiments, and they needed compute to run experiments, but the number of experiments they could run was limited by the number of human researchers they had, which is a scarce resource in the world, right? There's not a lot of people who can do AI research. Now, if AI itself can do AI research, the number of experiments we can run explodes, and therefore, the amount of compute you need for research also explodes. So we don't see a world where we will have unused to best compute for the foreseeable future, right? When I was referring to surprises, my worry is more on the downside of we are not able to actually build all the comp…

AI assessment note: “my worry is more on the downside of we are not able to actually build”

Q And, uh, again, without going into anything, uh, confidential, although I guess when you guys were public, all of this, we assume all of this will be public, but like, is that thousands of people at this stage? I mean, is that, uh, is, is that, uh, like multiple different teams? Or do you guys, like, outsource a bunch of things and works with a bunch of contractors?

A Uh, it's a portfolio approach, right? So we are never going to be in a world where, uh, We outsource everything or build anything ourselves, right? It's always going to be mixed because that's the, that's the reason, that's the, uh, uh, reasonable thing to do, right? So you don't want to put your rights in all, all in one basket. So we will have hyperscalers, products probably providing a big chunk of our compute, a majority of our compute. Um, we will have new clouds, uh, parts of our portfolio. Uh, we will be partnering with design, build firms that can build the compute that we need. And of course, we, we build some of it ourselves. And, and so we're always going to have a portfolio approach because at the scheme which we need, um, we will need to tap into all sources of compute. We can't just rely on one particular mechanism.

AI assessment note: “It's always going to be mixed because that's the reasonable thing to do”

Q your perspective on a, on a, on a spectrum where, um, You know, on the one hand, one extreme, you'd say, well, uh, the, uh, AI industry and computer industry has a PR problem, and there's no problem. It's just the, the, the problems that we cannot explain it well enough to, at the other extreme, actually, those communities have a point. Uh, what do you think the reality is?

A I think any time there's new technology, which is as, uh, as evolutionally as this technology is, Uh, there is always disruption that's gonna happen. Uh, but NFW, we have learned this over history that, uh, this always leads to better outcomes for society, right? And so, how do we draw a line from where we are today to that outcome, right? And, uh, explain to the world why this is the trajectory we all need to be on. I mean, it's our responsibility. To do that. Uh, on the community side, there's a little bit of a local versus a global issue. On the communities, I think data centers are, even today, a net positive to every community. Because we are building these data centers in rural areas of America, for example, right? Where there's nothing else that is being built on this scale. So we show up in rural Texas, we build a data center that Produces new property tax receipts for the community. That funds schools, that funds hospitals. We show up and we invest in new grid infrastructure, which otherwise would never have happened, because there's no demand. So there's a modernized grid that, that AA can enjoy. Basically we produce jobs.

AI assessment note: “data centers are, even today, a net positive to every community.”