Davis: AI cloud customers are forced into unwanted 3-year GPU contracts
Jared Quincy Davis · No Priors Ep. 77 | With Foundry CEO and Founder Jared Quincy Davis · Aug 22, 2024 · at 15:37
Jared Quincy Davis, founder of AI cloud company Foundry, explains the mismatch between GPU cloud provider terms and user needs on No Priors.
0:00 / 0:39exact quote · 39.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“In AI cloud today, you're kind of forced to get really long-term reservations often three years for a fixed amount of capacity. [944] Jared Quincy Davis: No one really wants 64 GPUs for three years or a thousand GPUs for three years. [949] Jared Quincy Davis: You want 10,000 for, you know, a couple of months, maybe nothing for a little while. [954] Jared Quincy Davis: maybe you don't know how much you need because you, you're launching a new product and you're not sure how much demand there will be for it. [960] Jared Quincy Davis: in how much inference capacity you'll need, etc. [962] Jared Quincy Davis: It's very, very challenging if you have to reserve your, you know, the total amount that you may need up front for a long duration.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jared Quincy Davis
Davis: Large-scale GPU pre-training utilization is sub-80% and often below 50%
“And so even for a lot of the more sophisticated orgs running large pre-trainings at scale,
The utilization sub-eighty percent, sometimes less than 50%, actually, depending on how bad of a batch they have and the frequency of failure in the cluster.”
Davis: Current AI cloud is basically co-location, not real cloud
“I think current AI cloud is not cloud in the originally intended sense by any means.
[627] Jared Quincy Davis: so we should pull on that thread, but I'd say right now it's basically co-location.
[631] Jared Quincy Davis: Yeah, it's basically co-location, right…”
Davis: Public clouds own only basis points of global GPU capacity
“What percentage of the world's GPU petaflock capacity, or exaflock capacity, is kind of owned by the major public clouds. And I've asked this many people, and I typically have gotten guesses, you know, in the high tens of percents, and the only time I got a lo…”
Davis: NVIDIA H100 GPU utilization is 25% or lower
“By many measures, utilization of these, even H-one hundred systems are kind of state-of-the-art, the most viable, the most precious, et cetera, Is, in my case, it's 20%, 25% or lower, according to some you know, pretty high quality data I've seen from some gre…”
Davis: Future AI infrastructure will rely far less on massive GPU clusters
“I think this actually points the way towards like what the infrastructure future might look like. And I think it looks a lot less like everything requiring these big clusters.”
Davis: AI workloads are shifting from large pre-training to batch inference
“I think people are getting more sophisticated at thinking about cost in a more of a life cycle way, and that's actually leading to the workload shifting from large pre-training more and more towards things like batch inference. Which is actually a really, real…”