The Exchanges

Q Let's start by talking about where energy consumption comes from in a typical data center. Can you, like, walk me through the pie, I guess, as I think about it, of energy consumption in a data center historically? And then you can tell me whether this new generation of AI data centers is any different from that perspective. But let's just start with, like, give me the breakdown.

A So, uh, typically the data center, uh, most of the consumption is in the servers, uh, now that data centers have gotten so efficient. Um, you know, historically, uh, there's a metric called PUE that, uh, looked at data center efficiency and what percentage was consumed by the data center backroom and what percentage was, uh, consumed by the servers. And at one time that was like, Two to three times the power was consumed in the back room, and then with the advent of, uh, PUE, people really focused on making data centers much more efficient, so now only about 10 to 20% of the power in a data center is consumed by the back room, so all the mechanical cooling and all that stuff. And then the remainder is consumed by all the equipment that deals with the data, the network, servers, and then all of the components within them. And so, uh, if you look at the, the pie within the data realm, the small percentage though it's growing is the network, you know, maybe 10% now or so, but it's continues to grow as networks get more and more complex. And then you look at the power conversion in The servers and the data equipment, it's like five percent of the total, and then the CPUs traditionally have been maybe up to 50%, and then the memory is 30%, and so that's approximately, uh, what you would find with the typical Pi.

AI assessment note: “CPUs traditionally have been maybe up to 50%, and then the memory is 30%”

Q And getting everything closer together has, it's like dual benefits, right? You get, you get better performance and less energy consumption because the further things are apart, the more basically wiring you need to connect them.

A But it's, uh, it's not as linear as it was. Before, if you got things closer, there's another effect called Denard scaling, which, uh, As you made chips, as the feature sizes got smaller on chips, the power for that same size would stay the same, but that's no longer the case anymore. So now, even though you're miniaturizing and integrating everything together, the power isn't necessarily going down. If you look at processors generation after generation, the power pretty much stayed the same, even though it had more performance. Now what you're seeing is As you're integrating more and more and getting more and more into the multi chip module, the power keeps going up generation after generation. Um, and so that's a kind of a new dynamic that's really driving a faster need for power to go up. At least that's my perspective.

AI assessment note: “now, even though you're miniaturizing and integrating everything together, the power isn't necessarily going down”

Q fundamental constraint on the growth of AI right now is one of, combination of one of three things. Uh, demand, like are there actual use cases that make sense? That could be one. Two is chips, like Nvidia's sold out for years, and there's only so many chips in the world, and three is energy. If you had to pick, like, which is the current, which places the current ceiling?

A Well, it's resources. I think, I think we're moving into a world where it's not It's not demand driven, it's supply driven. You know, and, and we talk about power, but I think it's more than just power. It's trades to build data centers, right? I remember there was a time where I was building a data center in Virginia and we consumed a, uh, 200 mile radius of all of the electricians in the region in Southern Virginia, um, had to pay overtime for them to travel. Uh, and that was for 30 megawatts. Now we're talking gigawatt scale data centers. Well, If, if we consumed all the electricians in with 30 megawatts, how big is that radius when you're talking three gigawatts or a hundred times the size? I mean, I think there's all these things. Power is the popular one right now. And there's no question that without power, you can't land a data center, but you can't build a data center either. If, if it's in the middle of nowhere, you're going to have to build cities along with it too. There's, it's, it's, uh, And you have to have the factories to build all the equipment. I mean, if we're really going to grow at the rate At what the demand seems to be looking like, you have to scale the whole supply chain. It's not just power. Power is, it's the one that people are worried about because of the long time scale.

AI assessment note: “it's not demand driven, it's supply driven. You know, and, and we talk about power”

Q Before we go, I know you've also been putting some thought into this, like, question of higher level, um, data center sustainability and talking about nature positive data centers. Can you just explain a little bit sort of what you're thinking there and what's possible?

A Yeah. So, you know, I've always been really big on efficiency. You know, I, I don't know if you know this, but PUE, I came up with PUE back 25 years ago. And so I was always kind of obsessed with efficiency. Um, and I think the industry has done a lot, uh, in terms of, uh, improving efficiency. And then of course, I've been very involved in dealing with CO two emissions and how could we do better with that? And so. But it's not just about carbon. I had this little wake-up call in when I was reading a National Geographic article where it showed the number of insects that were captured in a two-week period in Germany back in the seventies or something, or eighties. And then, 27 years later, they did the same thing, and they, it was like a 10th of the number of insects were captured in the same period. And it really, uh, concerned me. And so in my R and D team, I hired biologists because I do believe there's this responsibility we have with technology to actually integrate it with, uh, nature. Um, after seeing this, cause I'm looking at it going, well, if you know, the food triangle or whatever, if the bottom is disappearing and the top's growing with human population, that's ecosystem collapse. So my belief now is, uh, uh, looking at ecosystem health. So it's about carbon. It's about soil health. It's about, uh, water quality. It's about all these different things and these diffe…

AI assessment note: “change the design so that we increase pollinators and habitat for pollinators”

Q that could end up being Smaller, because as you said, you know, we, it still is way easier to cite a 20 megawatt or, I don't know, you know, smaller than that data center than it is to cite, or even to cite 50 20 megawatt data centers than to cite a gigawatt data center. So, you know, what is it about this, this paradigm that makes that so difficult?

A I think, and again, this is probably beyond my scope, but I think it really has to do with the size of the models and the latency, right? It takes too much time for, uh, light to travel larger distances, as hard as that is to believe. Um, and so they tend to concentrate it. But, you know, to your point, I do think there's an opportunity for innovation in how these things are architected, right? Um, I actually think constraints drive innovation. You know, when we talked about the fact that Moore's law has given a free ride for the, uh, compute industry, right? Now imagine if there's constraint and it's coming. And so people are rethinking architectures. GPUs have evolved out of the fact that, um, that constraints were coming with Moore's law, right? You had to rethink and create specialized, um, uh, specialized kind of chips instead of General purpose CPUs so that they could hyper-optimize and provide, continue the path of improved performance year over year. Um, so, so I think that's what kind of drives everything.

AI assessment note: “I think it really has to do with the size of the models and the latency”

Q developed a new chip, and it is, you know, orders of magnitude, potentially, more energy efficient. I think NVIDIA, when they released the Blackwell chip, said it's like 25 X more energy efficient. When they say stuff like that, first of all, what are they, what does it mean? Like, what are they actually referring to? Is there a metric that's traditional or is it, is it, does it vary?

A It's, it's, it's probably a performance metric, right? The performance per watt, and they are getting substantially improved performance per watt. But the thing is, uh, I remember seeing this IBM commercial, uh, 15 years ago or so, where IBM says someone walks into the data center and it's empty, except there's one machine. And, and IBM's argument is you buy my one machine, your, your whole operation collapses into one, right? One machine, and it's super efficient. But the reality is there's this thing called Javon's paradox, which, which is really about if a resource gets cheaper, you consume more of it. Right. And it generally, uh, applies to fuel or energy costs and so on. But, but if you really look at it, compute is a resource. So just because something's gotten so much more efficient, In fact, getting it very efficient may actually drive more use of it to where it'll increase and not decrease. And so I think there's a much broader view you have to take when someone says something's more efficient, it's, and they say it's going to solve all your problems and we're not going to have this power increase because everything's going to be solved with this, in this case, as you're saying, the 25 X. I think what, what happens is the cost of compute goes down. And when the cost of compute goes down, applications that weren't viable as a business before, now all of a sudden become …

AI assessment note: “It's, it's, it's probably a performance metric, right? The performance per watt”

Q power into the individual data center. And obviously from an energy perspective, that total amount of power, power requirement is, is probably the most important thing because citing a. 50 megawatt data center is probably infinitely easier at this point than citing a gigawatt data center. So. Do you have a sense of why that is? Like, what is it about this AI world that requires this concentration of power?

A So, I mean, that has always happened, right? Data centers have always required more and more power. The whole notion of Moore's law is the doubling of transistors on the same area every two years has given much more compute power as time goes on. And, and frankly, it's actually made the industry complacent. Not much has changed over the years in the past six decades. If you look at it, cause Moore's law pretty much gave performance improvements for free. The laptop you would buy, you will, you would always buy a new laptop. Nothing's really changed in the laptop other than you get better performance CPUs. And that's what you kept buying year over year. So not much changed for quite a long time. So when you start looking at where we're going now and now GPUs, GPUs are actually multi-chip modules now, right? Because what's happening is Moore's law is starting to lose its ability to improve. And the interesting thing is because now the feature sizes on the die are getting to the point where it's atomic scale. I think now they're getting close to two nanometer, uh, feature sizes. And if you look at the atom, a silicon atom, it's a 10th of that, so.

AI assessment note: “Moore's law is starting to lose its ability to improve”