Aug 22, 2024 · 42m · no-priors
No Priors Ep. 77 | With Foundry CEO and Founder Jared Quincy Davis
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Foundry founder and CEO Jared Quincy Davis joins Sarah Guo and Elad Gil on No Priors to discuss reimagining AI cloud infrastructure from first principles. He details how modern GPU hardware bottlenecks and loss of elasticity can be solved through intelligent orchestration, while exploring the industry's architectural transition toward Compound AI Systems.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 17.1% of the talking time here. How this is scored →
The hosts as informed peer 4.4 Guest teaching 4.8 Guest disagreement 0.7 The hosts pushing back 0.7
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 10:13 Jared rejects the premise of modern AI cloud
Jared forcefully rejects the notion that modern GPU providers are true clouds, arguing they have devolved into simple co-location facilities.
Hardest push from the hosts ▶ 12:19 Elad challenges Jared on early cloud adoption skepticismElad politely but firmly counters Jared's claim that early cloud had no believers, citing his personal experience founding startups in 2006 that saw AWS as an immediate breakthrough.
Biggest teaching moment ▶ 4:27 Jared explains GPU cluster failure dynamics and healing buffersJared explains the physical realities of modern DGX systems, clarifying why failure cascades necessitate keeping 10 to 20 percent of GPUs idle as healing buffers.
The host holds their own ▶ 16:34 Sarah maps out the compute abstraction continuumSarah demonstrates deep technical authority by systematically deconstructing the compute evolution from on-prem closet servers to serverless, diagnosing exactly where AI infrastructure currently lags.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Inspirations from AlphaFold and ChatGPT: Democratizing Compute Leverage | 2 | 4 | 1 | 0 | Sarah opens with a prompt about Foundry's genesis. Jared explains the asymmetrical compute advantage behind AlphaFold 2 and ChatGPT, reframing the David vs Goliath narrative around computational leverage. | |
| Foundry's AI-Native Cloud Infrastructure and Economic Advantages | 5 | 6 | 1 | 1 | Elad categorizes three different types of GPU cloud users to probe true utilization. Jared provides an in-depth breakdown of hardware failure rates and the necessity of reserving a 10 to 20 percent healing buffer in clusters. | |
| The 'Large Regime' and Distributed Systems Networking Challenges | 5 | 6 | 1 | 1 | Elad asks whether high failure rates stem from QC issues or architectural complexity. Jared introduces his definition of the 'large regime' where model weights exceed single-node memory, turning model execution into a distributed systems problem. | |
| Historical Cloud Paradigms and the Loss of Elasticity in AI | 7 | 5 | 2 | 5 | Jared asserts that early cloud computing had few initial believers and that current AI cloud is merely co-location. Elad pushes back based on his firsthand experience building startups in 2006, arguing startups instantly saw AWS as magic while enterprises hesitated. | |
| Evolution of Infrastructure Abstractions and AI Market Immaturity | 7 | 3 | 0 | 0 | Sarah demonstrates strong domain expertise by articulating the historical evolution from on-prem closet servers to colo, hosting, virtualization, and serverless, contrasting it with AI cloud's regression to rigid reservations. | |
| Foundry's Spot Usability Engine and the Parking Lot Analogy | 2 | 6 | 1 | 0 | Sarah asks about Foundry's latest product launch. Jared uses an extended parking lot analogy to explain spot usability engines and automated preemption management on GPU clusters. | |
| Global Compute Distribution and the MARS Resiliency Suite | 6 | 5 | 1 | 1 | Jared quizzes the hosts on peak Ethereum GPU capacity. Elad correctly estimates tens of millions of V100 equivalents and connects it to historical Bitcoin mining compute exceeding Google data centers. | |
| The Transition from Monolithic Pre-Training to Compound AI Systems | 5 | 7 | 1 | 0 | Sarah inquires about the strategic viability of alternatives to massive monolithic clusters. Jared explains the paradigm shift toward compound AI systems, horizontal scaling, and test-time compute exemplified by AlphaCode-2 and Llama 3. | |
| Research on Compound AI Systems and Verifiable Task Bootstrapping | 5 | 6 | 0 | 0 | Elad brings up Jared's recent research paper on compound AI system design. Jared explains verifiable task bootstrapping, showing how parallel candidate generation and verification achieved a 10x improvement in prime factorization. | |
| Extending Compound Architectures to Open-Ended Tasks and Future Directions | 4 | 5 | 0 | 0 | Sarah prompts Jared on applying compound architectures to open-ended tasks. Jared describes multi-model ensembling across frontier LLMs combined with heuristic verifiers and simulators. | |
| Episode Conclusion and Channel Information | 0 | 0 | 0 | 0 | Standard podcast outro with channel information and subscription links. |
Statements from this episode (20)
Davis: DeepMind built AlphaFold 2 with a team of just 18 people
“I think that one of the things that was so remarkable to me about AlphaFold II is, initially it was a really small team, you know, three, and then later, 18 people or so, and they solved what was kind of a fifty-year grand challenge in biology, which is a pret…”
Davis: OpenAI Had 400 Employees and $13B of Compute When Launching ChatGPT
“In OpenAI's case, you know, there were only 400 people, but had thirteen billion dollars worth of compute, you know, which is quite a bit of computational scale there.”
Davis: Democratizing AI Compute Will Increase AlphaFold-Level Breakthroughs 10x to 100x
“I think it would increase the frequency of events like AlphaFold II by 10 X, a hundred X, or maybe even more super linearly.”
Davis: AI teams hold 10% to 20% of GPUs idle as healing buffer
“And so one of the consequences of that is that it's very common now to hold aside 10 to 20% minimum of the GPUs that a team has as buffer, as healing buffer, in case of a failure so you can slot something else in to keep the training workload running, right?”
Davis: Large-scale GPU pre-training utilization is sub-80% and often below 50%
“And so even for a lot of the more sophisticated orgs running large pre-trainings at scale,
The utilization sub-eighty percent, sometimes less than 50%, actually, depending on how bad of a batch they have and the frequency of failure in the cluster.”
Davis: Probability of a GPU supercomputer running weeks without failure is zero
“And so because you have millions, perhaps, of individual components in this supercomputer, the probability that it will run for weeks on end, and this is basically a verbatim quote from Jensen's keynote is basically zero.”
Davis: NVIDIA's Mellanox buyout is among the best acquisitions in history
“And their acquisition of Mellanox was one of the better of all time, arguably, from a market cap creation perspective.”
Davis: Current AI cloud is basically co-location, not real cloud
“I think current AI cloud is not cloud in the originally intended sense by any means.
[627] Jared Quincy Davis: so we should pull on that thread, but I'd say right now it's basically co-location.
[631] Jared Quincy Davis: Yeah, it's basically co-location, right…”
Davis: AI cloud customers are forced into unwanted 3-year GPU contracts
“In AI cloud today, you're kind of forced to get really long-term reservations often three years for a fixed amount of capacity.
[944] Jared Quincy Davis: No one really wants 64 GPUs for three years or a thousand GPUs for three years.
[949] Jared Quincy Davis: …”
Davis: AI compute market lacks futures and hedging mechanisms found in commodities
“The markets aren't mature enough that there's any analog. So what we have in other domains, like in commodities markets like wheat, oil, et cetera, where you can buy options and futures and hedge and sell back and things like that. It's kind of a pretty, Still…”
Davis: Public clouds own only basis points of global GPU capacity
“What percentage of the world's GPU petaflock capacity, or exaflock capacity, is kind of owned by the major public clouds. And I've asked this many people, and I typically have gotten guesses, you know, in the high tens of percents, and the only time I got a lo…”
Davis: OpenAI trained GPT-3 on 10,000 V100s for 14.6 days
“GPT-III was trained on 10,000 V-one-hundred GPUs in an interconnected cluster in Azure for about 14.6 days.”
Davis: Peak Ethereum mining equaled 10 to 20 million V100 GPUs
“The very peak, the tippy top of Ethereum. How many V-One hundred equivalents were there, given there were 10,000 for two weeks for GPT-III? ... It was about 10 to twenty million.”
Davis: iPhone 15 Pro has more FP16 FLOPS than an NVIDIA V100
“Actually, an iPhone 15 pro now is actually stronger than a V-one hundred, as a funny example. It has about 35 teraflops in every 16, I believe where a V-one hundred is around 30”
Davis: NVIDIA H100 GPU utilization is 25% or lower
“By many measures, utilization of these, even H-one hundred systems are kind of state-of-the-art, the most viable, the most precious, et cetera, Is, in my case, it's 20%, 25% or lower, according to some you know, pretty high quality data I've seen from some gre…”
Davis: Future AI infrastructure will rely far less on massive GPU clusters
“I think this actually points the way towards like what the infrastructure future might look like. And I think it looks a lot less like everything requiring these big clusters.”
Davis: AI workloads are shifting from large pre-training to batch inference
“I think people are getting more sophisticated at thinking about cost in a more of a life cycle way, and that's actually leading to the workload shifting from large pre-training more and more towards things like batch inference. Which is actually a really, real…”
Davis: Compound AI approach increased prime factorization accuracy from 3.7% to 36.6%
“And so, you know, we did kind of some preliminary investigations here, and we were able to, in one case, a prime factorization, you know, kind of 10 x the performance, go from 3.7% to 36.6%. On prime factorization, which is pretty hard, kind of factor, you kno…”
Davis: Compound AI approach yielded a 3% MMLU performance bump
“The MMLU performance bump was about three percent. And to put that in perspective, the gap between some of the previous best models is often less than one percent. Between, for example, Gemini, 1.5, and Lama 3.1, and things like that. So, actually, 2.8% or thr…”
Davis: AI systems will make millions of heterogeneous model calls per question
“I think that what we'll see people doing is kind of composing, this sounds funny, but massive networks, Where maybe each stage in the network will basically be maybe some best of A, best of K component with many, many calls to different language models, you kn…”