GPU

“But a GPU, it has intrinsic value. Like, you can actually use that asset for something that's really, really valuable. And it doesn't matter where that GPU is. It could be in Iceland. It's still useful to anybody all over the world, as long as it's networked. …”

“We're very willing to pay a hundred K per month extra to get them earlier.”

“The most expensive part of a delivery of a GPU is if it's late, I'm still paying rent for that data center space.”

“It is a mistake to price optimize and to spend the money to buy it versus to rent it because I get you on the optimization, but you're not saving 10 times more. It's .5 X more per year.”

“Our engineers, really what I'm looking for is 10 really good decisions per day, which is very tiring, not like optimizing the random parts of the code. And each one has like, you know, five to 18 agents running at any point in time, doing long horizon tasks on…”

“It's already solved my dad's prostate cancer, because I figured out with a bunch of help from other people how to use GPUs to identify where in his body the lesion was.”

“And in fact, as you get to the smaller footprints, where more and more AI is going to take place, that's going to be a sweet spot for Arm, because the CPU's table stakes anyway, you have to have it to do all the things that are required in the edge device. But…”

“Open source is good for compute salespeople. If you're selling GPUs, you want everyone else's margin to be lower, so yours can be higher.”

“We're running you know, frontier level, one of the most intelligent models, right? OpenAI's largest, most capable, most intelligent model right now at 14 times faster than their normal, you know, GPU speeds.”

“I think my reaction when I see pictures like this is that it's very impressive graphics design. But I also don't see them building anything beyond just, or trying to build something better than just, you know, a traditional GPU, right? You know, they've made c…”

“They, I, for sure, I don't think they will generate free cash flow anytime soon. I think they're gonna generate a lot of operating cash flow, and then they'll use that to buy a lot of, you know, GPUs XPUs, whatever, whatever I'm gonna call them.”

“This is like, we're flat out, and people are reselling GPUs for [514] Ben Horowitz: Four times what they bought them for”

“By the way, it is so bad that right now if we have new companies going for GPUs, it's often in Mexico or Australia or in other country just because it is so difficult in the United States.”

“The power quality coming out of a diesel generator tends to be very dirty in terms of the voltage waveforms for your AC. And if you have spiky loads, it tends not to respond very well, so you have a lot of voltage sag or over voltage. So all that is very chall…”

“So lean still has a lot of shortcomings there. It's CPU based and you know, it's not, Like, getting that onto the GPU has a lot of nuances there. So, you know, a lot of work needs to be done. So what we've started with is a framework, you know, making that mor…”

“To our surprise, we found that it's not only, you know, accurate, it's almost as close to the what the traditional weather models can do accurately, but also tens of thousands of times faster. So what would take a big supercomputer to run can now be run. And w…”

“The GPU is fundamentally a throughput machine. The GPU is happiest when you give it a lot of work to do and let it chew through that work at peak utilization of its compute units.”

“There's a fundamental trade off on the GPU between being throughput oriented or latency optimized. And everyone has chosen latency optimization because the shape of usage was chatbot oriented.”

“Cerebris, Grok and a couple others that are coming out of stealth now, I think have made a very interesting bet on not just building another GPU, but actually building a different kind of accelerator that focuses on a different memory hierarchy. They want to m…”

“Cerebris and Grok and maybe a couple others, you should think of them as accelerators. What they are really good at is being used in conjunction with a more traditional GPU-like device that critically has this off-chip memory built in.”

“That operation runs at, you know, 70, 80% of peak utilization, and it's limited not by software, but by power. The way NVIDIA quotes peak flops is a little optimistic. You never hit that because of power throttling”

“I, look, Asics chips are really ideal if you're thinking about model customization. If you're saying, look, we're at a new phase in, in, in this AI build out, or what we really want to do is, is, is do a lot of model specialization. You don't need a GPU for th…”

“So you have, like, on the calls, you have both Andy Jassy and Cyanadella are out there saying, look, we're just building data centers. Like, these are the shells. We might not use them now. Maybe we'll use them in the future. And we only buy GPUs when we know …”

“The neural network, the benefit of neural network is, it's made up of a lot of standard GMM, general matrix multiplication, which, you know, is a famous, the GPU. GPU is highly optimized for this kind of operation. But the selection tree, you know, the old gen…”

“So we have some GPUs that are failing inside of modern data centers on an 18 month cycle, some that are failing on a much longer period.”

“The failure rates of GPUs used so intensively for training purposes are much higher than inference specific usage.”

“These layers are pretty costly and like that kind of limits what you can do with the architectures. They're not like, Not only are they like costly in terms of compute, they're just like not efficient on modern GPUs either. You have small hidden dimensions, la…”

“Serving large language model is a fundamentally different problem. Because serving it requires to run it on accelerators like GPUs or TPUs, and it is a computationally intensive process that will require a lot of engineering and ensuring that for each request,…”

“Probably BERT. And before that, it was like ResNet for computation, like images, computer vision classification. So ResNet already need to run on NVIDIA K-eighty, which is kind of one of the first SEU on AWS and other places. And, but way over, but even at thi…”

“AlexNet, First, you know, kind of, like, neural network to run on, on GPUs that we care about ran on two GPUs. And that's not like there are no missing decimal points or commas in there. Literally two.”

“The current status quo of we're spending two hundred billion dollars a year on CapEx. And, you know, yes, we're renting our GPUs, so revenue is accelerating, but fundamentally there's no There's not a new business line coming out of it. That is not sustainable…”

“You can, in a lot of cases, get slightly better outcomes at half the cost. But again, that half the cost, I think a lot of people hear that, they're like, that's bad for AI demand. It's actually not at all because the cost the user pays has, you know, is just …”

“Effectively, all models today are MOE models that are, you know, at least all models large enough that you would care to parallelize them across multiple GPUs.”

“Of course, Jensen is in some sense self-serving with this letter, because the more open source models are developed, the more companies are going to be training on GPUs. They're going to be fine tuning on their own data. And it's just more and more spend. It d…”

“Companies are spending on it, usually within Frontier Labs, at about 10 to 20% about the amount that they're spending on GPUs.”

“That's exactly one of the areas where the data layer, where you can clearly see that the data layer is where value is going to accrue. Because the GPUs Are the same GPUs in both cases. The problem is that, that data infrastructure, the flywheel, the data colle…”

“What I see today that's prominent in AI space is what I consider to be the CPU of intelligence unit. You have this one language model that's really large, that's very smart, that can do very complex reasoning tasks. That's like CPU. What I see coming and what …”

“I think, ah, you're gonna see more and more, ah high performance and low energy inference hardware systems, because I think everyone is now realizing that inference is the key to making, you know, these agent-based systems be available to more and more people,…”

“The moment that you start basically going to lower concurrency because you want better interactivity and better latency, Right? The performance the throughput drops. And it drops very sharply because all of a sudden you have a lot of, like, smaller kernels, yo…”

“Say, if you run Gemma four, On your GP, you might get like a hundred tokens per second if you have a fast card. If you run it in their cloud, you get anywhere from 800 to 1500 tokens per second. So call it 10 X faster.”

“And so we were trying to think about like a lot of things such that if the revenue growth took longer, To materialize than we thought it might, we could have, you know, consumer apps and media and all these other things that could help us monetize the GPUs tha…”

“In inference in AI, it's the exact opposite. You move a huge amount of data, all the weights, from memory to compute, and you need one calculation to generate the next word. And then you have to do it again. So all the time is dominated by the movement of data…”

“The JPs have a huge failure rate, so I'm sure you guys have spoken about this. Infant mortality is enormous, and they fail all the time.”

“And so the speed of moving waits to compute is about two and a half thousand times faster here than on a Wilben GP.”

“We've designed a fully distributed system. We run across five, six data center regions globally, and tap into scattered GPUs, and they are able to run massive jobs, our jobs.”

“Now with these agentic workflows, you actually need both. In fact, the ratio of CPU to GPU is becoming like one to one.”

“And for reinforcement learning, it's always somewhere, like, around five to, like, six percent. So, said differently, that means that we're getting, like, five percent of the actual GPU computing power that we're paying for.”

“Now if you invest one thousand billion dollars today into GPUs for data centers, this means that within five years you are going to lose nine hundred billion dollars, you know. Somebody is going to lose nine hundred billion dollars in the near future because t…”

“And the paradox is like right now they can name their price, but that isn't where we're going to be in five years. You can argue about how quickly the infrastructure gets built out and how fast the GPUs arrive, blah, blah, blah, blah. Fine. But that's a supply…”

“They had available capacity because the Grok model wasn't used very much. So they had these GPUs that were sitting around and that's a bad idea. And so they sold a whole block of them or released a whole block of them to Anthropic.”

“There it's not about scaling from zero to one, but it's how do we scale really elastically from, like, thousand to 1500 GPUs very quickly in, in a given region.”

“That's why it's something we've invested a lot of time in is actually building our own reliability layer on top. So if the GPU falls off the bus or something happens, we user workloads are not affected.”

“Like the way you move around your KV cache and how efficiently you can do it, how efficiently you move your weights from your training GPUs to your inference GPUs in RL is, there's a lot of degrees of freedom, and it is basically a systems problem of Moving me…”

“And so GPUs and LPUs combined ended up giving better performance across the performance curves.”

“The LPU and GPU, as mentioned, they, they're better at different parts of what's, what's called the decoder layer of an LLM. The GPU is better at the attention portion, and the LPU is better at sort of applying the weights, which is the thing that gets trained…”

“In the second game, there was this famous move called move 37, which was creative. It was original. It actually wasn't completely original. It was a one in 10,000 game move. It had been in the canon of games that we're trained on. But when we went back and pla…”

“So over time, these GPUs have, you know, gotten, you know, as good and better than TPUs, you know, as the career of the TPU, I have to admit, GPUs are now better.”

“When LLMs first started to become a thing, I remember getting a phone call from the CEO of GitHub basically saying, I need a bunch of GPUs. We've now gotten LLMs to be able to do code completion, even though, you know, we're part of Microsoft and everything yo…”

“DLSS is our real-time AI for graphics, and it makes a small GPU run like a big GPU. It's about 10 times more efficient because rather than computing the color of every pixel for every frame, we use AI to infer the color.”

“Bitcoin miners run at under a quarter of the voltage of GPUs.”