Dec 5, 2024 · 40m · big-technology
AWS CEO Matt Garman on Amazon's Big AI Chips Bet, Working With OpenAI, and Nuclear Energy
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Recorded live at AWS re:Invent, AWS CEO Matt Garman sits down with Alex Kantrowitz to detail Amazon's multi-layered enterprise AI strategy, spanning custom Trainium 2 silicon, Bedrock's model-agnostic architecture, nuclear energy investments, and long-term cloud growth.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Alex holds 22.4% of the talking time here. How this is scored →
Alex as informed peer 5.7 Guest teaching 5.0 Guest disagreement 2.0 Alex pushing back 4.1
speaking balance: gold is Alex, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 4:39 Rejecting the competitor single-model cloud strategy
Garman directly dismisses the host's premise that Amazon is falling behind Microsoft and Google, calling the 'one model to rule them all' philosophy fundamentally mistaken.
Hardest push from Alex ▶ 2:52 Pressing why AWS has not built its own leading frontier modelKantrowitz confronts Garman on why AWS's unmatched compute and scaling capabilities have not yielded a state-of-the-art in-house model to rival OpenAI or Anthropic.
Biggest teaching moment ▶ 16:25 Explaining why modern LLM inference runs on training-class siliconGarman educates the host on modern LLM architecture, correcting the misconception that smaller Inferentia chips handle frontier models with hundreds of billions of parameters.
Alex holds their own ▶ 36:29 Quoting Andy Jassy's internal memo on corporate bureaucracyKantrowitz reads verbatim from Andy Jassy's memo detailing excessive pre-meetings and decision-making friction, pressing Garman to answer whether AWS suffered from cultural degradation.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Alex as informed peer | Guest teaching | Guest disagreement | Alex pushing back | Why |
|---|---|---|---|---|---|---|
| Data Center Scaling and Project Rainier with Anthropic | 6 | 5 | 2 | 4 | Kantrowitz opens by citing Elon Musk's 122-day Colossus buildout to challenge AWS's infrastructure leadership narrative. Garman counters by explaining AWS's focus on aggregate infinite scale and reveals the Project Rainier collaboration with Anthropic on Trainium 2. | |
| Amazon's Model-Agnostic AI Strategy and Bedrock Platform | 7 | 6 | 4 | 6 | Kantrowitz presses Garman on why AWS lacks its own frontier model despite having best-in-class infrastructure, contrasting them with Google and Microsoft's OpenAI exclusivity. Garman directly rejects the premise, framing single-model dependency as fundamentally flawed compared to customer choice. | |
| AWS Openness to Hosting OpenAI and Competitor Models | 6 | 5 | 2 | 5 | Kantrowitz drills into Bedrock's gaps without OpenAI or Google and presses Garman on whether AWS is actively talking to OpenAI. Garman explains the strategic rationale behind the $4B Anthropic partnership and the co-design of Trainium 2 silicon. | |
| Balancing the NVIDIA Partnership with Custom AWS Chips | 5 | 4 | 2 | 4 | Kantrowitz explores the tension of praising Nvidia while developing rival custom silicon and asks how AWS negotiates chip allocation against tech peers. Garman draws parallels to managing Intel, AMD, and Graviton ecosystems. | |
| Lowering AI Inference Costs and Automated Model Distillation | 6 | 6 | 1 | 3 | Kantrowitz highlights the high cost of inference that blocks enterprise POCs from reaching production. Garman clarifies that massive modern models run inference on training-class silicon like Trainium 2 and outlines automated model distillation. | |
| Evaluating NVIDIA Blackwell and Upcoming EC2 P6 Instances | 6 | 5 | 2 | 5 | Kantrowitz probes whether the current short list of generative AI use cases justifies trillions in market capitalization. Garman responds with AWS's Q Developer agents and biology AI examples, characterizing current tooling as merely the tip of the iceberg. | |
| AI Automation and the Evolution of Developer Skills | 6 | 6 | 3 | 5 | Kantrowitz raises concerns from developers about skill atrophy when relying entirely on AI coding assistants. Garman dismisses the fear using a mental math and Excel analogy, arguing foundational problem-solving remains unchanged. | |
| Amazon Bedrock Multi-Agent Collaboration and Complex Workflows | 5 | 5 | 1 | 2 | Kantrowitz connects AWS's agentic roadmap with comments made on the show by Salesforce's Marc Benioff. Garman details Bedrock's newly announced multi-agent collaboration system using a multi-variable store location planning scenario. | |
| Nuclear Energy Investments and Small Modular Data Center Reactors | 6 | 5 | 2 | 5 | Kantrowitz questions the industry's nuclear energy push by pointing out unresolved nuclear waste and safety concerns. Garman highlights modern safety improvements, Amazon's $500M X-Energy investment, and co-locating SMRs next to data centers. | |
| Cloud Cost Optimization Cycle and Resumed Enterprise Growth | 6 | 4 | 2 | 4 | Kantrowitz questions AWS's period of slowing growth and quotes Andy Jassy's internal memo about bureaucracy and pre-meetings. Garman acknowledges that AWS experienced bureaucratic drag and explains structural reforms to push ownership back to frontline teams. | |
| Long-Term Cloud Migration Opportunity and Market Acceleration | 4 | 4 | 1 | 2 | Kantrowitz asks what realistic ceiling exists for cloud workload migration beyond the current 20% baseline. Garman projects that cloud penetration will eventually invert to 80% or more over coming decades. |
Statements from this episode (17)
Garman: AWS S3 currently stores 400 trillion objects
“Today, S-Tree actually stores 400 trillion objects.”
Garman: AWS Project Rainier will deploy hundreds of thousands of Trainium2 chips in 2025
“We're building together with them what we call Project Rainier and so it's using our next generation Tranium II chips, and this cluster that we're building for them in twenty-twenty-five will be five times the size, the number of exaflops that they use to trai…”
Garman: Expecting one dominant AI model is a fundamentally flawed premise
“And this is where a lot of people started, and I just think it's fundamentally the wrong way of thinking about it. Which is a lot of times people are thinking about there's just going to be this one model and I want to have the one model that's going to be the…”
Garman: AWS is open to hosting OpenAI and Google Gemini models
“We're always open to having OpenAI be available in AWS someday, or having Gemini models be available in AWS someday, and maybe someday we will spend more time focused on our own models, for sure.”
Garman: Trainium 2 Delivers 30% to 40% Better Price-Performance Than GPUs
“We see 30 to 40% price performance games versus our instances that are GPU powered today.”
Garman: Past AI Accelerators Failed Due to Weak Software Versus NVIDIA
“I think that's one of the things where people who've tried to build Accelerated platforms before have fallen down is the software support has not been as good as Nvidia software support is fantastic.”
Garman: NVIDIA treats AWS very fairly in allocating GPU supplies
“They're very fair in dealing with us and we give long-term forecasts and they tell us what they can supply.”
Garman: Generative AI workloads are 50/50 today but shifting toward inference
“You know, we're still probably seeing about that ratio of fifty-fifty. I think more and more it's more inference than training, and increasingly we'll see more and more of the workload shift that way.”
Garman: AWS to launch Blackwell-based EC2 P6 instances in early 2025
“We're going to announce that the P six, which is the Blackwell based instance that's coming early next year.”
Garman: Nvidia Blackwell delivers about 2.5x compute performance of H100
“I think we were expecting about two and a half times the compute performance out of the Blackwell chip that you get out of an H-one hundred”
Garman: Developers only spend about one hour a day actively coding
“It turns out, also, developers, on average, code about one hour a day. The rest of their day is spent doing documentation. It's spent writing unit tests. It's spent writing, doing code reviews. It's spent doing, You know, going to meetings, it's spent doing up…”
Garman: AWS uses automated reasoning internally to verify IAM permission changes
“We actually use it internally to make sure that our permissioning system is actually, when you change permissions, that it's actually behaving as expected.”
Garman: AWS automated reasoning mathematically eliminates AI hallucinations in bounded domains
“So by that, by this kind of mechanism, you're, like, systematically able to actually mathematically prove that you got the right answer coming out of this and completely eliminate hallucinations for that area, right? It doesn't mean that we've eliminated hallu…”
Garman: Small modular nuclear reactors will be major energy sources starting 2030
“We do think that you know, over the next probably starting somewhere in, in 20, 30 and beyond these small modular reactors, which is what X-energy builds, are gonna be a huge component of this.”
Garman: Cloud Optimization Is Largely Finished, Funding AI and AWS Growth
“Customers, I think number one, a lot of them have been optimized, right? And there's only so much you can kind of squeeze into an Optimize place and customers are still looking for optimizations, but a lot of that work has been done and they're using some of t…”