GPU

“The future of AI and machine learning is going to be in GPUs.”

“The thing that is really expensive I mean, there are many things that can be expensive, but one of the things that is really expensive is I want to train a model from scratch that is very large and it's going to take me you know, low tens of people, probably 2…”

“TPUs are the most scalable architecture, like we have zero failure rate with our TPU language model training, whereas with GPUs it's like there's an ECC error”

“Transformers are mostly amazing because they can be trained on GPUs. There are lots of other models that we're currently not exploring because they're not trainable on GPUs.”

“I don't think NVIDIA in five years time will be sitting here proud as the only people who can do a GPU.”

“Right now if you know, in San Francisco, the big, you know, researchers and the big CEOs brag about how many GPUs they have. You know, they're the sort of the biggest indicator of how serious they are about AI is, is how many GPUs they have. But I think in the…”

“About half the cost of the data center is GPUs, half is everything else.”

“No one's ever going to train a frontier model on the same data center twice, because by the time you've trained it, the GPUs will be outdated, and the data center will be too small.”

“The 10,000 GPUs that we've now brought online this summer, you know, that came from this capital, allow us to make incredible advancements in model capabilities”

“Today, interconnecting more than 32,000 GPUs is extremely challenging.”

“What you're going to see now is now that everyone has seen this deep seek architecture, they're going to go great. I have hundreds of thousands of GPUs. I'm now going to use a lot of them to create a lot of synthetic data. And then I'm going to train the bejes…”

“More than five x lower. Just the memory alone in the latest GPUs costs more than our fully loaded CapEx per chip deployed.”

“Like, if you look at the last GTC, there was an announcement that the latest GPUs were 30 x faster than the previous generation. And when you look at how it was done, there was this curve that looked kind of like this, and then it Basically ended here, and the…”

“So we use about a third of the energy per token. About over a three-year period, one-third of our cost is the OpEx, which is mostly energy and data center rent, and two-thirds is the CapEx, which means that since we're one-third of the energy, the cost to run …”

“One of the concerns right now is about Malaysia or Singapore, that region over there being a place where people are deploying GPUs with the wink, wink, like we're not going to rent it to China, right? But that's a belief that a lot of people are doing that. Ot…”

“Actually, that's why Grok achieves, ah, not Grok, but Grok, Cerebras, and all these folks, they achieve very high performance single stream is because the data is right in the chip that doesn't have to get it from memory, which is slow, which GPU has to do.”

“I very much worry there will be an oversupply of these chips. The problem is, is that, you know, remember, the chips are the collateral. So, you know, somewhere, you know, in the US or whatever, there's going to be a data center with like a thousand GPUs that …”

“If you go from one GPU to two, you don't get twice the performance. Maybe you get 10% better performance. Yeah, that's the dirty secret nobody talks about. I'm talking inference, right? So, so you go from, let's say, a hundred to a 110 by doubling the amount o…”

“Fundamentally, GPUs cannot deliver, deliver this, plain and simple at scale.”

“GPUs are, you know, are a good trick for AI, but they're not built for AI.”

“The fundamental architecture of the GPU with off-chip memory is not great for inference. Now, they will continue to do well in inference, but it can be beaten, and I think they know it.”

“In a GPU, most of the time it's doing inference, it's five or seven percent utilized. That means it's 95 or 93% wasted.”

“The hard part with AI work is results and intermediate results have to be moved a lot. And therein is the most complicated part. They have to be moved to memory and from memory. And they have to be broken up and moved among GPUs.”

“They are making a bunch of architectural changes to GPUs to make them better and better inference.”

“And now it processes hundreds of billions of tokens of code a day, that single model. And it's running on our own GPUs.”

“I think that the two people who have, the two categories in the stack that didn't meaningfully exist at scale in SAS and cloud land that exists now are the GPUs, which is all Jensen and the models. Neither of those kind of categories even existed. And obviousl…”

“So, people look at the TPU as a big success, and what they don't realize is that there were about three chip efforts at Google at the same time and only one of them ended up outperforming GPUs.”

“Microsoft in one quarter deployed a bunch of GPUs, and then announced that they weren't going to make them available in Azure because they made more money using them themselves than renting them out.”

“You have to write a check two years in advance to get GPUs. For us, you write us a check for a million LPUs, and the first of those LPUs starts showing up six months later.”

“The idea that NVIDIA is unstoppable because of the software and hardware connection, because we have to have GPUs. I know there's a lot of truth to that, but literally as an end user, I went right back and forth to them today. No, no issue. TPUs, GPUs, LLMs, a…”

“That is a limitation for all GPUs, but not us. We don't use it.”

“We've designed a fully distributed system. We run across five, six data center regions globally, and tap into scattered GPUs, and they are able to run massive jobs, our jobs.”

“What I see today that's prominent in AI space is what I consider to be the CPU of intelligence unit. You have this one language model that's really large, that's very smart, that can do very complex reasoning tasks. That's like CPU. What I see coming and what …”

“That's exactly one of the areas where the data layer, where you can clearly see that the data layer is where value is going to accrue. Because the GPUs Are the same GPUs in both cases. The problem is that, that data infrastructure, the flywheel, the data colle…”

“Of course, Jensen is in some sense self-serving with this letter, because the more open source models are developed, the more companies are going to be training on GPUs. They're going to be fine tuning on their own data. And it's just more and more spend. It d…”

“Companies are spending on it, usually within Frontier Labs, at about 10 to 20% about the amount that they're spending on GPUs.”

“I, look, Asics chips are really ideal if you're thinking about model customization. If you're saying, look, we're at a new phase in, in, in this AI build out, or what we really want to do is, is, is do a lot of model specialization. You don't need a GPU for th…”

“Open source is good for compute salespeople. If you're selling GPUs, you want everyone else's margin to be lower, so yours can be higher.”

“But a GPU, it has intrinsic value. Like, you can actually use that asset for something that's really, really valuable. And it doesn't matter where that GPU is. It could be in Iceland. It's still useful to anybody all over the world, as long as it's networked. …”

“It's already solved my dad's prostate cancer, because I figured out with a bunch of help from other people how to use GPUs to identify where in his body the lesion was.”

“The most expensive part of a delivery of a GPU is if it's late, I'm still paying rent for that data center space.”

“We're very willing to pay a hundred K per month extra to get them earlier.”

“Our engineers, really what I'm looking for is 10 really good decisions per day, which is very tiring, not like optimizing the random parts of the code. And each one has like, you know, five to 18 agents running at any point in time, doing long horizon tasks on…”

“It is a mistake to price optimize and to spend the money to buy it versus to rent it because I get you on the optimization, but you're not saving 10 times more. It's .5 X more per year.”