Dec 5, 2023 · 1h 7m · latent-space
The State of Silicon and the GPU Poors - with Dylan Patel of SemiAnalysis
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Dylan Patel, Chief Analyst at SemiAnalysis, joins the Latent Space Podcast to examine the economics, physical bottlenecks, supply chain geopolitics, and silicon architecture underpinning modern AI, detailing how compute disparity shapes strategies for both hyperscalers and resource-constrained developers.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 11.5% of the talking time here. How this is scored →
The hosts as informed peer 4.7 Guest teaching 5.8 Guest disagreement 3.0 The hosts pushing back 1.7
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 30:55 Dismissing fine-tuning of smaller models
Patel rejects the conventional industry enthusiasm for fine-tuning 7B models, bluntly calling it a useless waste of time compared to utilizing larger models at decent batch sizes.
Hardest push from the hosts ▶ 31:44 Swyx challenges GPT-3.5 sizing and sparse routingSwyx presses Patel on the exact parameter scale of GPT-3.5 versus Falcon 140B and questions whether the model is smaller or sparsely routed.
Biggest teaching moment ▶ 17:35 Detailed derivation of LLM inference bandwidth requirementsPatel walks through the exact math showing why reading 70 billion parameters at human reading speed requires over 2 TB/s of bandwidth, exposing the inefficiency of standard inference libraries.
The host holds their own ▶ 30:07 Alessio and Swyx analyze model depreciation velocityAlessio and Swyx frame the economic dilemma of investing heavily in model training when open-source architecture advances make models obsolete within three months.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Introductions and the Rise of SemiAnalysis | 4 | 3 | 2 | 1 | Swyx introduces Dylan Patel and mentions his own background covering semiconductors. Patel gives a brief, polite correction noting he had discussed Mixture of Experts months earlier before it gained viral attention. | |
| The Economics of AI Infrastructure and Training Costs | 5 | 5 | 2 | 2 | Swyx references hedge fund semi coverage and quotes Patel's provocative thesis on training costs. Patel explains the shifting cost structure of AI software from high R&D to high COGS and infrastructure operating costs. | |
| The GPU Poor vs. GPU Rich Landscape | 3 | 6 | 5 | 1 | Patel delivers an extended analysis on GPU allocations, calling SF startup GPU boasting detached from supply realities. He forcefully criticizes Hugging Face benchmarks like TruthfulQA as garbage and dismisses batch size 1 inference on high-end GPUs. | |
| Google TPUs, Framework Ecosystems, and Compilers | 5 | 6 | 2 | 2 | Alessio brings up previous guest Chris Lattner and asks about PyTorch vs JAX/XLA on Google hardware. Patel details the distinction between TPU v5 and v5e and how lower-level compilers like Triton and Palace fit into the stack. | |
| Compute Utilization Metrics: MFU in Training vs. MBU in Inference | 4 | 8 | 4 | 1 | Swyx prompts Patel to define hardware utilization metrics. Patel gives an in-depth lecture on arithmetic intensity, contrasting training MFU with inference MBU, while calling existing open source inference implementations extremely inefficient. | |
| Interconnect Scaling and Google's Partnership with Broadcom | 5 | 6 | 2 | 2 | Swyx identifies networking as the slowest scaling dimension in hardware systems. Patel explains Broadcom's critical role in Google TPU co-design and why interconnect engineering makes custom silicon difficult for startups. | |
| Strategic Opportunities for GPU-Poor Developers and Startups | 5 | 6 | 2 | 1 | Alessio asks how under-resourced startups can compete in a compute-heavy ecosystem. Patel highlights edge inference, speculative decoding like Medusa, and asynchronous training architectures as practical avenues for the GPU-poor. | |
| Model Depreciation, Latency Profiling, and Noam Shazeer's Vision | 6 | 6 | 4 | 4 | Alessio and Swyx examine rapid model depreciation cycles. Patel argues fine-tuning smaller models is largely a waste of time and explains how to infer proprietary model architectures using Azure inference latency bounds. | |
| The AI Silicon Landscape and Nvidia's Formidable Moat | 5 | 8 | 5 | 2 | Alessio asks about alternative AI silicon startups and who is viable. Patel explains why first-wave startups bet wrongly on on-chip SRAM over off-chip DRAM bandwidth and breaks down why Nvidia's gross margin and release cadence create an insurmountable moat. | |
| Frontier Lab Partnerships and Apple's GenAI Conundrum | 5 | 5 | 3 | 2 | Patel predicts frontier labs will eventually drift apart from cloud partners due to scale ambitions. Swyx asks about Apple's AI positioning, and Patel explains how Apple's brand risk tolerance conflicts with shipping imperfect LLMs. | |
| AI Safety Debates and Semiconductor Supply Chain Fragility | 4 | 7 | 4 | 1 | Patel challenges security-by-obscurity safety stances before Swyx asks about rebuilding the semiconductor supply chain in the US. Patel delivers an unequivocal rejection, citing obscure geographic monopolies across Austrian tools and Japanese specialty chemicals. | |
| Lightning Round, Research Workflow, and the Genie Question | 5 | 4 | 1 | 1 | In the wrap-up lightning round, Patel recommends foundational reading and describes SemiAnalysis's rigorous on-the-ground supply chain research. He answers the genie question by highlighting multi-datacenter distributed training as AI's ultimate unsolved bottleneck. |
Statements from this episode (34)
Patel: SemiAnalysis reported GPT-4 mixture of experts architecture in January
“Just being clear, I talked about mixture of experts in January, it's just people didn't really notice it.”
Patel: AWS holds major infrastructure efficiency advantage over Azure and GCP
“Behind the scenes, they've done a lot on the infrastructure that is super custom, That Microsoft, Azure, and Google Cloud just don't even match in terms of efficiency. Like, if you think about the cost to rent out SSD space, so the cost to rent, you know, offe…”
Patel: AI software will have lower human R&D costs but higher COGS than SaaS
“An AI software looks like it's going to be very different in my opinion, right? Like the R&D cost is much lower in terms of people but the cost of goods sold in terms of actually operating the service, I think will be much higher, right?”
Patel: Frontier AI model training costs are effectively irrelevant
“In my opinion, I think that's a little bit spicy, but yeah, it's like training costs are irrelevant, right? Like GPT-IV, right? Like 20,000 A-one hundreds. That's like, I know it sounds like a lot of money.”
Patel: Nvidia manufactured 400k H100s last quarter and will sell 530k this quarter
“There's 400 to 500,000 being, 400,000 manufactured last quarter, and like five 30,000 this quarter being sold, right, of H-Hundreds”
Patel: Nvidia will sell over 3 million GPUs in 2024
“NVIDIA is going to sell well over three million, you know, total GPUs next year. You know, over a million H 100 this year alone, right?”
Patel: TruthfulQA is a garbage benchmark that gets gamed
“Truthful QA is a garbage benchmark, like, you, like, some of the models that are very high on there, if you use it for five seconds, you're like, this is garbage, right?”
Patel: Google will have more compute than any company by far
“Google is gonna have more compute than any other company in the world, period, by, like, a large, large factor.”
Patel: Google TPU v5e is half the size of TPU v5p
“TPUv-V-V-E is, like the new one, but it's mostly, mostly an inference chip. It's a small chip. It's a little bit, it's about half the size of a TPUv-V-V. That chip, you know, you can get very good performance on, like, Of Lama, 70 B inference, right?”
Patel: Hardware vendors and developers are coalescing around OpenAI's Triton
“Likewise, there's OpenAI's Triton, like, what they're trying to do there, and like you know, everyone's really coalescing around Triton. You know, people, you know, third-party hardware vendors”
Patel: AI inference will deploy more GPUs than training by 2024
“LLM inference will be bigger than training, or multimodal, whatever, blah, blah, blah inference will be bigger than training, you know, probably next year, in fact at least in terms of GPUs deployed,”
Patel: Hugging Face libraries achieve only 15% MBU for inference
“Hugging Face's libraries are actually very inefficient, like incredibly inefficient for inference. You get like, 15% MBU on, on, on, on some configurations, like eight, eight, eight, eight, eight, eight, 100, and LLAMA-seventy-beat, you get like, 15%, which is…”
Patel: Google is co-designing TPU v5 and TPU v6 with Broadcom
“TPU, you know, all the way through to all, TPU v-five, which is the one they're in production of now, and six, and, you know, all these are all gonna be you know, co-designed with Broadcom.”
Patel: Google targets 2027 to exit Broadcom partnership after 2025 attempt failed
“Their latest target to get away from Broadcom is twenty-twenty-seven, right? But like, you know, that's four years from now. Chip design cycle is four years. So they already tried to get away in twenty-twenty-five, and that failed.”
Patel: Broadcom is working on Meta's second-generation internal AI chip
“So Broadcom is, is, is, is working with, on Google's chip right now, but of course on Meta's, Meta's internal AI chip, which they're on the second generation of, working on that.”
Patel: AI chip bottlenecks lie in interconnects and memory, not matrix multiplication
“Multiplying tensors is kind of, you know, anyone can, there's a lot of people who've made good matrix multiply units, right? But it's about, like, getting good utilization out of those, and interfacing with the memory, and interfacing with other chips really e…”
Patel: Running LLaMA-70B at reading speed requires 2.1 TB/s memory bandwidth
“Hey, to run Llama's seventy billion requires two terabytes a second of memory bandwidth, 2.1, at reading, human reading speed.”
Patel: Google will avoid deploying local laptop models to retain control
“I don't think Google is going to deploy a model that I can run on my laptop to help me with code or help me with, you know, XYZ. They're always going to want to run it on the cloud for control.”
Patel: Several companies make tens of millions from adult AI models
“I think there's a couple companies who make, Tens of millions of dollars of revenue from, yeah, from LLMs or diffusion models for porn”
Patel: Fine-tuning existing small models for cloud use is useless
“Unless, unless you're fine tuning for on device use, I think fine tuning current existing models, especially the smaller ones is a useless waste of time, right?”
Patel: No open-source model is currently close to GPT-3.5
“And like, there's nothing open source that is anywhere close to 3.5 yet.”
Patel: Roughly 10 companies have compute to beat GPT-4 within six months
“There's, like, 10 companies that have enough compute in one single data center to be able to beat GPT-IV, right? Like, straight up, like, if not today, within the next six months, right?”
Patel: Intel will release a chip surpassing Nvidia H100 within a quarter
“Intel bought that company from him, and then shut it down, and bought this other AI company, and now that company is kind of, ah, you know, got new chips. They're gonna release a better chip than the H 100, ah, within the next quarter or so, right?”
Patel: AMD MI300 will beat Nvidia H100 on paper within a quarter
“AMD. They have a GPU. MI 300. That will be better than the H 100 in a quarter or so. Now, that says nothing about how hard it is to program it, but at least hardware-wise, on paper, it's better.”
Patel: Nvidia and Google control over 80% of advanced AI silicon manufacturing capacity
“To simplify it, NVIDIA has a little bit more than half, and Google has, like, 30%, right, through Broadcom. So it's like, the total capacity for everyone else is much lower, and they're all sharing it”
Patel: AMD MI300 costs more than twice as much to manufacture as Nvidia H100
“In the case of amd their manufacturing costs for mi-thruhundred or more than twice that of h-one hundred and it only beats h-one hundred by a little bit from you know performance stuff i've seen”
Patel: Nvidia to ship next-gen chip in Q2/Q3 2024 with 3x LLM performance
“Nvidia's releasing a new chip, you know, in, you know, they're gonna announce it in March, and they're gonna release it, you know, and ship it, you know, Q-two, Q-three next year anyways, right? And that chip will probably be three or four times as good. Right…”
Patel: OpenAI and Microsoft partnership will likely collapse within years
“Yeah, I expect in the next few years that the OpenAI and Microsoft probably falls apart too.”
Patel: Microsoft's custom AI chip performs worse than Nvidia H100
“Microsoft's going to announce their chip soon. It's worse performance than the H-H-E-N-H-E-D but the cost effectiveness of it is, is better for Microsoft internally, just because they don't have to pay the Nvidia tax.”
Patel: US and China can each shut down the other's chip supply chain
“The U.S. Could absolutely shut down the Chinese semiconductor supply chain, they won't, but, and China could absolutely shut down the U.S. One, actually, by the way.”
Patel: All sub-7nm chips depend on tooling from one Austrian monopoly
“There is no chip that is less than seven nanometer that doesn't get touched by this one Austrian company's tool, right? And there is no alternative.”
Patel: TSMC's Arizona fab still depends on Taiwan for masks and shipping
“TSMC is building a fab in Arizona. It's quite a bit smaller than the fabs in, in, in Taiwan, but even ignoring that, those fabs still have to ship everything to Taiwan back anyways. And also they have to get what's called a mask from Taiwan and get sent to get…”