NVIDIA H100

“So I was speaking with CoreWeave at the end of the year last year, beginning of the year this year, you know, around New Year time and they said they were actually renting out H-one hundreds for higher prices than they had previously.”

“You can see that with some of the prices that people are paying for each 100, right? Hoppers are, you know, kind of, you know, not, not obsolete, right? But they're a previous gen chip and nor in any normal situation, we're not totally supply constrained. No o…”

“You can see that with some of the prices that people are paying for each 100, right? Hoppers are, you know, kind of, you know, not, not obsolete, right? But they're a previous gen chip and nor in any normal situation, we're not totally supply constrained. No o…”

“The largest part of that cost structure is the depreciation that is associated with that GPU hour.”

“You actually look at the chips that we deployed in twenty-twenty-three, H-one hundreds. We're now leasing those out at a higher rate. Now than we were originally in 20, 23.”

“To just run the swarm, I think it's like a 16 node of H-one hundreds.”

“Like at this point H 100 pricing has massively firmed up. B 200 pricing definitely has super firmed up. And like, hey, there's clearly demand.”

“Because the H-Twenty was oddly enhanced. You know that it had less flops, but more memory. In fact, it had more memory than the H-One hundred”

“We found that it's great for flash attention to specifically, we were able to be H-one hundred. We also found that like the less time you spend in like dense compute, like the less time you spend in tensor cores specifically, or less time you spend in lower bi…”

“Yeah, my guess is it's probably on the order of 50 k h 100, something like that.”

“Their architecture led to a speed up of 7000 X compared to the NVIDIA Jetson Nano, 300 X compared with NVIDIA RTX 4090, and then a hundred X compared to the NVIDIA H 100. And the energy was reduced by 40,000 X compared to Jetson Nano, 90,000 X compared to RTX …”

“And if you look at the memory bandwidth on the H-Twenty, it actually has 20% more memory bandwidth than the H-One-Hundred.”

“And by the way, like the H 100 and A 100 kind of suck at encoding and decoding video, right? They're actually slower than like a T four, for example, which has like media sort of like drivers and stuff that can actually like do it really quickly.”

“You need, I think 671 gigabytes for the weights, and you also need an extra memory for the KV cache, so it's not possible to run that on H-one hundred.”

“If every, you know, American adult used half an H 100 as, You know, a form of co-pilot for their, you know, daily workflows and, you know, social interactions, et cetera. Like, that would require 250 gigawatts of power, right?”

“By many measures, utilization of these, even H-one hundred systems are kind of state-of-the-art, the most viable, the most precious, et cetera, Is, in my case, it's 20%, 25% or lower, according to some you know, pretty high quality data I've seen from some gre…”

“A 100 in particular are pretty available. Obviously the AMD chips that we also agnostically work with the MI 300 and MI two fifties, those are available. H 100 still kind of. A little bit harder to get, but you can get started very easily with any of those oth…”

“For a long period of time, we had, 500 A-One hundreds, because we made a commitment, like and they were constantly being delayed, I think, because of H-One hundred, supply demand, whatever, like, reasons that and it was also very hard to get, like, a lot of co…”

“I think customers are still struggling with availability for the most premium chips. And I think, you know, whether that's eight, 108, 100, I think even when there is availability, you're oftentimes looking for like three to six weeks of negotiating with cloud…”

“One of the problems with NVIDIA was that they released their H 100, but they didn't really reduce its cost. You know, it is, it's two is, you know, 1.9 X faster, but two X costlier.”

“That is by the end of this year, and 600 K H-One hundred equivalents. With 250 K H-one hundreds and including all of the other GPU or accelerator stuff, it would be 600 and something K aggregate capacity.”

“NVIDIA is going to sell well over three million, you know, total GPUs next year. You know, over a million H 100 this year alone, right?”

“Intel bought that company from him, and then shut it down, and bought this other AI company, and now that company is kind of, ah, you know, got new chips. They're gonna release a better chip than the H 100, ah, within the next quarter or so, right?”

“AMD. They have a GPU. MI 300. That will be better than the H 100 in a quarter or so. Now, that says nothing about how hard it is to program it, but at least hardware-wise, on paper, it's better.”

“And then NVIDIA is also part of the round because we have a very large GPU cluster of 10,000 H 100, and it's very helpful to have NVIDIA as part of that.”

“The challenges, the GPUs there are like, not the ones that AI folks want to use, right? Like kind of all the AI is really zeroed in on like, how do we get a 100 or H 100 and the GPUs that like folks used for Ethereum mining and like similar is like older ones …”

“I mean, people using. Eight h, 100 to do inference on a big model. I mean, that's. Half a million dollars.”

“The constraint today is actually in how much high-performance memory is available on the chip. These models need to be in memory all at the same time, and they take up hundreds of gigabytes. So while memory has scaled up, I mean, we're gonna get flashing all t…”

“And by NVIDIA bifurcating their chip architectures into a gaming segment that does not have this latest COWAS technology, this allows them to monopolize, like, a huge amount of TSMC's capacity to make the COWAS chips, specifically for these H-one hundreds, whi…”

“So they launched it in September, twenty-twenty-two. It's the successor to the A-One hundred. One GPU, one H-One hundred, cost 40,000 dollars.”

“So, the reason you want an H-one hundred is they're 30 times faster than an A-one hundred, which mind you is only like two and a half years older. It is nine times faster for AI training.”

“You can get access to a DGX server. That's eight A 100 for about 30 bucks an hour, or you can go over to AWS and get a P five dot 48 X large instance, which is eight H 100, which I believe is an HGX server for about a hundred dollars an hour.”

“For 90% of most companies model training use cases, the tiny box will be five X faster for the same price.”