Sep 6, 2023 · 2h 54m · acquired
Nvidia Part III: The Dawn of the AI Era (2022-2023) (Audio) · Acquired
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
This episode of Acquired analyzes Nvidia's monumental rise during the generative AI revolution, detailing how its twenty-year convergence of GPU hardware, the CUDA software ecosystem, and full-stack data center architecture enabled the company to achieve a trillion-dollar valuation. Co-hosts Ben Gilbert and David Rosenthal dissect the technological breakthroughs, strategic maneuvers, and competitive moats that established Nvidia as the indispensable computing foundation of modern artificial intelligence.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Ben and David hold 99.8% of the talking time here. How this is scored →
Ben and David as informed peer 7.9 Guest teaching 1.0 Guest disagreement 0.2 Ben and David pushing back 0.2
speaking balance: gold is Ben and David, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 19:30 Pushing back on the social media AI monopoly straw man
Ben gently checks David's theory that Snap and Musical.ly failed to reach Facebook scale purely because Google and Meta monopolized AI talent, calling it an overextended straw man.
Hardest push from Ben and David ▶ 1:49:15 Correcting China export regulation characterizationBen pushes back on David's assertion that the Biden administration enacted an outright ban on advanced computing to China, noting specific export performance thresholds.
Biggest teaching moment ▶ 56:30 Ben's masterclass on von Neumann bottleneckBen walks step-by-step through assembly instructions and memory bus constraints to explain why traditional CPU architectures fail at large-scale AI parallel workloads.
Ben and David hold their own ▶ 2:46:10 The insurmountable moat checklistBen delivers an exhaustive technical breakdown demonstrating why matching Nvidia requires simultaneous parity in GPU silicon, NVLink, InfiniBand, CoWoS fab capacity, and 10,000 person-years of CUDA software.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Ben and David as informed peer | Guest teaching | Guest disagreement | Ben and David pushing back | Why |
|---|---|---|---|---|---|---|
| Introductory Banter and Episode Theme Song | 6 | 1 | 0 | 0 | Ben and David open the episode framing the dramatic 18-month shift in Nvidia's market positioning and generative AI landscape. They review prior predictions from their 2022 episodes collaboratively with high shared research expertise. | |
| AlexNet and the Dawn of GPU Computing | 8 | 2 | 0 | 0 | David and Ben recount the 2012 AlexNet breakthrough on ImageNet, breaking down how convolutional neural networks run on consumer GeForce GPUs using CUDA initiated accelerated computing. | |
| The Big Tech AI Duopoly and Feed Algorithms | 8 | 2 | 1 | 1 | David explains the post-AlexNet acquisition of Hinton's lab by Google and the talent duopoly formed with Meta. Ben gently challenges David's speculative idea about Snap and TikTok's AI access being the sole differentiator. | |
| The Rosewood Dinner and OpenAI's Founding | 8 | 1 | 0 | 0 | The hosts detail the famous 2015 Rosewood dinner hosted by Sam Altman and Elon Musk to recruit top researchers away from Google and Meta, leading to the formation of OpenAI. | |
| Early Language Models and Next-Token Prediction | 8 | 1 | 0 | 0 | Ben details Andrej Karpathy's early insights on language models and unsupervised next-token prediction, while David quotes Ilya Sutskever's detective novel analogy explaining predictive understanding. | |
| The Transformer Revolution: Attention Is All You Need | 9 | 1 | 0 | 0 | Ben delivers a thorough explanation of the 2017 Google 'Attention Is All You Need' paper, explaining O(n^2) computational complexity and how transformer parallelism unlocked GPU scaling. | |
| Large Language Models and Emergent Scaling | 8 | 1 | 0 | 0 | The hosts explore parameter scaling from GPT-1 to GPT-4 and emergent reasoning capabilities resulting from training massive transformer models on parallel GPU clusters. | |
| OpenAI's For-Profit Shift and Microsoft Alliance | 8 | 1 | 0 | 0 | David and Ben discuss Elon Musk's departure from OpenAI, the creation of the capped-profit entity, and Microsoft's landmark multibillion-dollar investments and cloud partnership. | |
| Sponsor: Statsig | 6 | 0 | 0 | 0 | Ben and David present a dedicated sponsor segment for Statsig, detailing feature flagging, model parameter tuning, and experimentation in AI workflows. | |
| Computer Architecture and the Von Neumann Bottleneck | 9 | 1 | 0 | 0 | Ben provides a deep-dive lecture into classic von Neumann computer architecture, the memory-compute bus bottleneck, and why parallel stream architectures are mandatory for AI workloads. | |
| Nvidia's Hardware Trifecta: Hopper, Grace, and Mellanox | 9 | 1 | 0 | 0 | David and Ben detail Nvidia's multi-year hardware roadmap: the Mellanox acquisition bringing InfiniBand, the Grace ARM CPU, and the bifurcated Hopper data center architecture. | |
| Advanced Semiconductor Packaging and TSMC CoWoS | 9 | 1 | 0 | 0 | Ben and David examine TSMC's 2.5D CoWoS packaging, explaining reticle size physical limits and why packaging capacity is the primary supply bottleneck for H100 production. | |
| The H100 GPU and DGX Supercomputer Solutions | 8 | 1 | 0 | 0 | The hosts break down the specifications and pricing economics of the H100 GPU and DGX SuperPod systems, emphasizing bundle margins and full-stack integration. | |
| AI Model Compression and the Launch of DGX Cloud | 8 | 1 | 0 | 0 | Ben compares LLM training to lossy image compression, and David explains the strategic launch of DGX Cloud hosted across third-party hyperscalers. | |
| Historic Q2 FY24 Financials and Data Center TAM | 8 | 1 | 0 | 0 | David and Ben analyze Nvidia's unprecedented Q2 FY24 financial results, discussing the $10.3B data center quarter and Jensen's re-framed $1 trillion data center TAM. | |
| Sponsor: Blinkist and Go1 | 6 | 0 | 0 | 0 | The hosts deliver a sponsor read for Blinkist and Go1, detailing their curated AI reading list and executive learning benefits. | |
| Gross Margins, China Regulations, and Omniverse | 8 | 1 | 1 | 1 | Ben notes rising 70%+ gross margins and clarifies US export controls versus outright bans on Chinese shipments, while David discusses modified A800/H800 chips and Omniverse enterprise potential. | |
| Nvidia's Corporate Culture and Jensen Huang's Leadership | 7 | 1 | 0 | 0 | David and Ben discuss Nvidia's flat corporate culture, Jensen Huang's 40 direct reports, and his total dedication to work without outside distractions. | |
| Seven Powers Analysis of Nvidia's Competitive Moat | 9 | 1 | 1 | 1 | The hosts apply Hamilton Helmer's 7 Powers framework to Nvidia, debating counter-positioning and rigorously analyzing CUDA's 10,000 person-years of scale and network economies compared to iOS. | |
| Acquired Playbook: The Apple of Enterprise AI | 8 | 1 | 0 | 0 | Ben and David articulate the Acquired Playbook takeaways: Nvidia as the Apple of Enterprise AI, vertical system integration, and patient market timing. | |
| The Bear Case for Nvidia | 8 | 1 | 1 | 1 | The hosts thoroughly evaluate bear cases including big tech custom silicon, PyTorch abstraction, market overhype cycles, and shifting compute from training to inference. | |
| The Bull Case for Nvidia | 8 | 1 | 0 | 0 | David and Ben outline bull arguments: ubiquitous accelerated computing, massive data center CAPEX refresh, and TSMC's 50% compound growth forecasts for AI hardware. | |
| The Competitor Checklist and Closing Reflections | 9 | 1 | 0 | 0 | Ben summarizes the full checklist of technical, manufacturing, networking, and software hurdles a rival would need to overcome to displace Nvidia. | |
| Carveouts and Episode Conclusion | 6 | 0 | 0 | 0 | The hosts wrap up with personal media carveouts (Alias and Moana), listener thank-yous, and community announcements. |
Statements from this episode (64)
ChatGPT reached 100 million active users faster than any application
“First, with OpenAI's ChatGPT, which became the fastest app in history to a hundred million active users”
Nvidia projected a $1 trillion TAM by capturing 1% of global industry
“In one of Nvidia's earnings slides in 2021. They put up their total addressable market and they said they had a one trillion dollar TAM. And the way that they calculated this was that they were going to serve customers who Provided a hundred trillion dollars w…”
AlexNet reduced ImageNet mislabeling error rates from 25% to 15%
“I think the error rate went from mislabeling images 25% of the time to suddenly only mislabeling them 15% of the time, and that was like a huge leap over the tiny incremental progress that had been made along the way.”
AlexNet was trained on two consumer Nvidia GeForce GTX 580 GPUs
“And what these guys from Toronto did is they went out probably to their local Best Buy or equivalent in Canada. They bought two GeForce GTX-Five-Eighty's, which were the top-of-the-line cards at the time, and they wrote their algorithm, their convolutional neu…”
Parallel computing on GPUs acts as an Archimedes lever on Moore's Law
“Whatever advances are happening in Moore's law and the number of transistors on a chip, If you have an algorithm that can run in parallel, which is not all problem spaces, but many can, then you can basically lever up Moore's Law by hundreds of times, or thous…”
AI pioneer Geoffrey Hinton descends from mathematicians George and Mary Boole
“He is the great-great-grandson of George and Mary Boole.”
AI feed recommendations made Instagram a $100B to $500B asset for Meta
“Instagram would have been a great acquisition anyway, but it was AI-powered recommendations in the feed that made that into a 102 105 hundred billion dollar asset for Facebook.”
OpenAI was founded to reach artificial general intelligence before major tech companies
“The founding of OpenAI was motivated by the desire to find AGI or Artificial General Intelligence first before the big tech companies did.”
Ilya Sutskever left Google in 2015 to co-found OpenAI
“So after the dinner, Ilya leaves Google and signs up to become, as we said, co-founder and chief scientist of a new independent AI non-profit research lab backed by Elon and Sam. OpenAI.”
Language structure and knowledge are embedded directly within raw text data
“The very structure of language and the way to interpret knowledge is actually embedded in the training data itself rather than requiring labeling.”
Ilya Sutskever argued next-token prediction inherently requires true world understanding
“The more accurately an LLM predicts that next word, i.e. The name of the criminal. Ipso facto, the greater its understanding, not only of the novel, but of all general human-level knowledge and intelligence, because you need all of your experience in the world…”
Transformer attention computational cost scales quadratically with input prompt length
“Traditionally, you'd say this is very, very inefficient, and it actually means that the larger your context window, aka token limit, aka prompt length, gets, the more computationally expensive it gets on a quadratic basis. So doubling your input means quadrupl…”
Transformers enabled parallel training of sequence models on GPUs
“So the big innovation here is you could now train sequence-based models In a parallel way. You couldn't train models of this size at all before, let alone cost effectively.”
Google possesses strong technical AI capabilities but lacks fundamental product sense
“I mean, this is Google here. Like, the capabilities are there. The product sense. Not as much.”
Training early large transformer models was financially untenable for anyone except Google
“GPUs and NVIDIA and the transformer made it possible, but to work with the size of models you're talking about spending an amount of money that's certainly for a nonprofit and anybody really except Google was untenable.”
GPT parameter counts grew from 120 million in GPT-1 to 1.7 trillion
“GPT-I had roughly a hundred and twenty million parameters that it was trained on. GPT-II had 1.5 billion. GPT-III had a hundred and seventy five billion, and GPT-IV, OpenAI hasn't announced, but it's rumored that it has about 1.7 trillion parameters that it wa…”
Scaling compute and data created unpredicted emergent reasoning in language models
“We don't change anything about the structure, we just give it way more data and let it Run these models for a long time and make the parameters of the model way bigger, and like, no researchers expected them to reason about the world as well as they do, but it…”
Google built TPUs to offset compute costs while still buying Nvidia hardware
“Even Google at this point in time, this is when they start building their own chips, TPUs, because, you know, they're still buying tons of hardware from NVIDIA, but they're also starting to source their own here.”
Generative AI and cloud compute convergence is Nvidia's greatest growth tailwind
“And the combination of those three things turns out to be basically the single greatest moment that could ever happen for NVIDIA.”
Predominant enterprise AI GPU compute consumption will occur via the cloud
“It looks like the predominant way that companies are going to use that compute is gonna be in the cloud.”
Nvidia spent five years building platforms to replace Intel's x86 data centers
“NVIDIA has literally just spent the past five years working insanely hard to build a new computing platform for The data center, a GPU accelerated computing platform to, in their minds, replace the old CPU led Intel dominated x-eighty six architecture in the d…”
Nvidia bypassed the von Neumann bottleneck via massive hardware parallelism
“So the magical unlock, of course, is to make a computer that is not a von Neumann architecture. To make programs executable in parallel and massively increase the number of processors or cores. And that is exactly what NVIDIA did on the hardware side, and all …”
On-chip memory capacity is the primary hardware bottleneck in AI training
“The constraint today is actually in how much high-performance memory is available on the chip. These models need to be in memory all at the same time, and they take up hundreds of gigabytes. So while memory has scaled up, I mean, we're gonna get flashing all t…”
Semiconductor chips cannot be etched larger due to extreme ultraviolet photolithography limits
“Due to a quirk of the extreme ultraviolet photolithography that we talked about, the EUV on the TSMC episode, chips are already the full size of the reticle. It's a physics and wavelength constraint. You really can't etch chips larger without some new inventio…”
Nvidia's $7 billion acquisition of Mellanox is one of tech's best ever
“They made one of the best acquisitions of all time back in 2020, and nobody had any idea. They bought a quirky little networking company out of Israel called Mellanox.”
Nvidia split consumer and data center GPU architectures in September 2022
“Up until NVIDIA's current GPU generation, the hopper generation of GPUs for the data center, there was only one GPU architecture at NVIDIA, and that same architecture and those same chips from the same wafers made at TSMC, some of them went to consumer gaming …”
Nvidia bifurcated its architecture to monopolize TSMC's CoWoS packaging capacity
“And by NVIDIA bifurcating their chip architectures into a gaming segment that does not have this latest COWAS technology, this allows them to monopolize, like, a huge amount of TSMC's capacity to make the COWAS chips, specifically for these H-one hundreds, whi…”
Chip-on-Wafer-on-Substrate advanced packaging accounts for 10% to 15% of TSMC's capacity
“So COOS represents right now about 10 to 15% of TSMC's capacities, and many of the facilities are custom-built for exactly these types of chips that they're producing.”
Nvidia sells DGX artificial intelligence server systems for $150,000 to $300,000
“NVIDIA sells these DGX systems for like, a 150 to 300,000 dollars a box.”
Nvidia launched the H100 GPU in September 2022 retailing at $40,000
“So they launched it in September, twenty-twenty-two. It's the successor to the A-One hundred. One GPU, one H-One hundred, cost 40,000 dollars.”
Nvidia's H100 GPU is nine times faster for AI training than A100
“So, the reason you want an H-one hundred is they're 30 times faster than an A-one hundred, which mind you is only like two and a half years older. It is nine times faster for AI training.”
Nvidia is using artificial intelligence to design its own semiconductor chips
“They're actually using AI to design the chips themselves now.”
Cloud instances cost $30 hourly for A100s and $100 hourly for H100s
“You can get access to a DGX server. That's eight A 100 for about 30 bucks an hour, or you can go over to AWS and get a P five dot 48 X large instance, which is eight H 100, which I believe is an HGX server for about a hundred dollars an hour.”
Nvidia DGX Cloud starting prices are $37,000 monthly for an A100 system
“So starting price for DGX Cloud is 37,000 dollars a month, which will get you an A-one hundred based system, not an H-one hundred based system.”
DGX Cloud rental pricing yields a three-month capex payback on A100 hardware
“A listener helped us out and estimated that the cost to actually build an equivalent A-one hundred DGX system Would be today something like a 120 K. Remember, this is the previous generation. This is not H-one hundreds. And you can rent it for 37 K a month. So…”
Cloud service providers account for half of Nvidia's data center revenue
“The CFO Colette Kress said on their last earnings call that about half of the revenue from the data center business unit is CSPs. And then I believe after that is the consumer internet companies, and after that is enterprises.”
Nvidia's Q2 FY24 release was one of history's greatest corporate earnings reports
“I think this was one of, if not the most incredible earnings release by any scaled public company ever. Seriously, no matter what happens going forward, last week was a historic moment.”
Global data centers contain one trillion dollars worth of installed hard assets
“There is one trillion dollars worth of hard assets sitting in data centers around the world right now.”
Global annual capital expenditure to expand and update data centers is $250B
“Annual spend on data centers to Update and add to that CapEx is two hundred and fifty billion dollars a year.”
OpenAI is reported to have exceeded a $1 billion annualized revenue run-rate
“ChatGPT made it so OpenAI is rumored to be doing over a billion dollar run rate now. Maybe multiple single digit billions, and still growing, meaningfully.”
Nvidia's CUDA platform reached four million registered developers by May 2023
“If you look at the number of CUDA developers over time, it was released in 2006, It took four years to get the first 100,000 people. Then by twenty-sixteen, 13 years in, they got to a million developers. Then just two years later, they got to two million. So 1…”
Nvidia functions fundamentally as a platform company akin to Microsoft
“They do make semiconductors, and they do make data center gear, but really they are a platform company. The right analogy for NVIDIA also is Microsoft. They make the operating system. They make the programming environment. They make many of the applications.”
Nvidia released the Megatron transformer model shortly after acquiring Mellanox in 2019
“In March of 2019, NVIDIA announced they were acquiring Mellanox for seven billion dollars in cash... In August of 2019, NVIDIA released what was at the time the largest transformer-based language model called Megatron. Eight point Three billion parameters trai…”
Nvidia achieved 70% gross margins in Q2 FY2024 and forecasted 72%
“This last quarter, they had a gross margin of 70%, and they forecasted for next quarter to have a gross margin of 72%.”
Nvidia's gross margins will not erode significantly below 65% as shortages subside
“That's gonna go away, but I don't think this very high, you know, 65% plus margin is gonna erode too much.”
Nvidia is the only viable platform for training GPT-class frontier AI models
“If you want to train GPT or a GPT class model, there's one option. You're doing it on NVIDIA. There's one option.”
Sales to mainland China accounted for 25% of Nvidia's 2022 total revenue
“Last year, China was 25%, or sales to mainland China, was 25% of NVIDIA's revenue.”
Microsoft employs five times more people per market-cap dollar than Nvidia
“They have 26,000 employees, and that sounds like a big number, but for comparison, Microsoft, whose market cap is only twice as big, has 220,000. So that is five X the number of employees per dollar of market cap going on over at Microsoft”
Nvidia generates $46 million in market capitalization for every single employee
“They have forty six million dollars of market cap per employee.”
Nvidia operates with no designated active succession plan for CEO Jensen Huang
“I don't think there's anyone else there where they're, like, getting ready for that person to take over. I think The company is a extension of Jensen's thoughts and will and drive and belief about the future, and that's kind of what happens.”
AMD lacks Nvidia's TSMC advanced packaging capacity and CUDA developer ecosystem
“AMD doesn't have all this capacity reserved from TSMC, at least not for the 2.5 D packaging process for the high end GPUs. AMD doesn't have the developer ecosystem from CUDA.”
Competitive moats erode and margins compress as total addressable markets become massive
“Every moat only works if the castle is sufficiently small. If the prize at the end of the finish line becomes sufficiently large, you're gonna need a bigger moat, and you need to figure out a you know, how to defend the castle harder.”
Nvidia accelerated to a six-month product release cycle following the COVID-19 pandemic
“Since COVID, NVIDIA has re-accelerated to a six month shipping cycle. They've been doing two GTCs a year, most years since COVID, which is insane for the level of technology complexity that they're doing.”
Nvidia has an installed base of 500 million CUDA-capable GPUs globally
“Today there are five hundred million CUDA capable GPUs for developers to target.”
Enterprise buyers face zero career risk choosing Nvidia hardware over unproven competitors
“Nobody is getting fired for buying Nvidia anytime soon.”
Most AI workloads run adequately on A100s unless training GPT-4 class models
“Most workloads can be run on A-Hundreds unless you're doing model training of GPT-IV.”
Nvidia will eventually operate its own complete data center buildings
“They used to have to plug into other people's servers, and then they started making servers that plugged into other people's racks and rows and architectures, and then they started making their own entire rows and walls, and at some point here, they're gonna s…”
Microsoft is the exclusive cloud infrastructure provider for OpenAI
“Microsoft is the exclusive cloud infrastructure provider for OpenAI, which runs, as far as we know, solely on NVIDIA infrastructure, but they buy it all through Microsoft.”
A cloud provider's traditional market share will dictate their AI era dominance
“I bet the way it plays out is that where you landed in cloud one point O strongly dictates where you will land in this AI cloud era.”
Data center GPUs represent a $30 billion market growing to $50 billion
“Right now, GPUs In the data center are like a thirty billion dollar a year market going to like a fifty billion dollar next year”
The AI market faces an investor crisis of confidence within 18 months
“I think there's a pretty reasonable chance that there's some falter in the next 12 to 18 months where there's a crisis of confidence among investors where at some point something will come out Where we all observe, oh, maybe GPTs aren't as useful as we thought…”
TSMC expects its AI hardware revenue to grow 50% annually through 2028
“TSMC, in their last earnings, said that AI hardware currently only represents six percent of their revenue, but all indications over there is that they expect AI revenue to grow 50% per year for the next five years.”
A large portion of machine learning inference will move to edge devices
“I suspect a lot of inference will get done on the edge. If you think about the insane amount of compute that's walking around in our pockets that is not fully leveraged right now, there's gonna be a lot of machine learning done on phones that are gonna, like, …”
Any challenger unseating Nvidia's AI dominance requires an unforeseen flank attack
“So I think the bottom line here is it nearly impossible to compete with them head on, and if anybody's gonna unseat NVIDIA in the future of AI and accelerated computing, it's either gonna be from some unknown flank attack that they don't see, or the future wil…”