Davis: AI systems will make millions of heterogeneous model calls per question
Jared Quincy Davis · No Priors Ep. 77 | With Foundry CEO and Founder Jared Quincy Davis · Aug 22, 2024 · at 40:39
Jared Quincy Davis, founder and CEO of Foundry, discusses the future of compound AI architectures with hosts Sarah Guo and Elad Gil.
0:00 / 0:44exact quote · 44.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think that what we'll see people doing is kind of composing, this sounds funny, but massive networks, Where maybe each stage in the network will basically be maybe some best of A, best of K component with many, many calls to different language models, you know, Claude, Gemini, GPT-IV, each with their own, you know, spikes in terms of capabilities. And you'll kind of throw multiple of them at questions in many cases, and then kind of choose the best response. You might, you know, also ensemble that with other components like classical heuristic-based systems and, you know, simulators, et cetera. And kind of compose large networks that may make millions of calls to answer a question. And I think that type of approach, it sounds kind of farcical right now, but I think it'll seem common sense pretty soon.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jared Quincy Davis
Davis: Large-scale GPU pre-training utilization is sub-80% and often below 50%
“And so even for a lot of the more sophisticated orgs running large pre-trainings at scale,
The utilization sub-eighty percent, sometimes less than 50%, actually, depending on how bad of a batch they have and the frequency of failure in the cluster.”
Davis: Current AI cloud is basically co-location, not real cloud
“I think current AI cloud is not cloud in the originally intended sense by any means.
[627] Jared Quincy Davis: so we should pull on that thread, but I'd say right now it's basically co-location.
[631] Jared Quincy Davis: Yeah, it's basically co-location, right…”
Davis: AI cloud customers are forced into unwanted 3-year GPU contracts
“In AI cloud today, you're kind of forced to get really long-term reservations often three years for a fixed amount of capacity.
[944] Jared Quincy Davis: No one really wants 64 GPUs for three years or a thousand GPUs for three years.
[949] Jared Quincy Davis: …”
Davis: Public clouds own only basis points of global GPU capacity
“What percentage of the world's GPU petaflock capacity, or exaflock capacity, is kind of owned by the major public clouds. And I've asked this many people, and I typically have gotten guesses, you know, in the high tens of percents, and the only time I got a lo…”
Davis: NVIDIA H100 GPU utilization is 25% or lower
“By many measures, utilization of these, even H-one hundred systems are kind of state-of-the-art, the most viable, the most precious, et cetera, Is, in my case, it's 20%, 25% or lower, according to some you know, pretty high quality data I've seen from some gre…”