May 1, 2024 · 41m · big-technology
AWS VP of AI and Data Swami Sivasubramanian — GenAI's Growth Potential
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
AWS Vice President of AI and Data Swami Sivasubramanian joins Alex Kantrowitz to examine whether generative AI is approaching practical limits in compute, data, and power. Sivasubramanian outlines AWS's multi-layered strategy to sustain AI progress through custom Trainium silicon, efficient model architectures, Amazon Bedrock, and tailored enterprise solutions.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Alex holds 26.3% of the talking time here. How this is scored →
Alex as informed peer 4.9 Guest teaching 5.1 Guest disagreement 2.0 Alex pushing back 3.8
speaking balance: gold is Alex, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 13:20 Swami firmly rejects Kantrowitz's characterization
Swami immediately counters Alex's assertion that he is ignoring frontier AI evolution, clarifying that customer application customization is where actual enterprise value and step-change innovation occur.
Hardest push from Alex ▶ 13:01 Alex directly challenges the enterprise customization pivotAlex openly tells Swami he disagrees with his premise, asserting that tuning on Rocket Mortgage data does not advance the frontier of AI capabilities.
Biggest teaching moment ▶ 28:20 Swami details compiler-augmented reasoning systemsSwami demystifies LLM reasoning by explaining that practical reasoning in tools like Amazon Q requires interleaving language models with external compilers and feedback loops rather than relying on pure LLM generation.
Alex holds their own ▶ 8:03 Alex details Llama 3 compute clusters and data exhaustionAlex demonstrates deep reporting knowledge by citing Meta's massive cluster scale and internal considerations of purchasing book publishers to challenge the limits of traditional scaling.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Alex as informed peer | Guest teaching | Guest disagreement | Alex pushing back | Why |
|---|---|---|---|---|---|---|
| Energy Constraints and AWS Data Center Sustainability | 5 | 4 | 2 | 6 | Alex pushes Swami repeatedly to address Mark Zuckerberg's point on energy constraints rather than allowing him to pivot purely to compute chip innovation. Swami acknowledges energy is a real bottleneck while highlighting AWS's focus on data center sustainability and custom silicon. | |
| Overcoming Data Limits with Enterprise Customization and Bedrock | 6 | 5 | 3 | 4 | Alex cites reporting on Meta running out of training data and considering buying book publishers to ask if traditional scaling hits a wall. Swami counters that the enterprise future lies not in ever-larger generic models, but in customizing smaller, highly efficient models through Bedrock. | |
| The Debate Over Foundational Model Evolution vs Enterprise Impact | 6 | 4 | 4 | 7 | Alex openly disagrees with Swami's framing, arguing that enterprise tuning on mortgage data does not advance frontier AI capabilities. Swami defends his position, stating that step-change productivity gains for individual enterprises are equally transformative. | |
| Model Efficiency and Smaller High-Performance LLMs | 4 | 6 | 1 | 2 | Alex asks how smaller models can outperform previous-generation larger models. Swami provides a technical explanation covering token density, reinforcement recipes, and emerging non-transformer architectures like state-space models. | |
| AWS AI Strategy and the Anthropic Partnership | 4 | 4 | 1 | 2 | Alex asks about the strategic rationale of AWS building internal LLMs while heavily backing Anthropic. Swami explains AWS's long-standing customer choice philosophy dating back to specialized databases, emphasizing that complex AI applications require multiple specialized models. | |
| Custom AI Silicon: Trainium and Inferentia Chips | 5 | 5 | 2 | 3 | Alex probes the competitive viability of Trainium against Nvidia's CUDA ecosystem and mentions Groq's fast inference. Swami outlines Trainium2 and Inferentia cost-performance specs, while politely declining to comment in depth on Groq. | |
| The Mechanics of AI Reasoning and Amazon Q | 4 | 7 | 1 | 1 | Alex inquires about the mechanics of frontier model reasoning. Swami delivers an extensive breakdown of SWE-bench, step-by-step plan generation in Amazon Q, and why real-world reasoning relies on coupling LLMs with deterministic tools like compilers. | |
| Cloud vs On-Device LLMs and Hierarchical Inference | 5 | 6 | 2 | 5 | Alex cites Apple's open-source OpenELM models to ask if on-device inference threatens the cloud computing model. Swami educates on hierarchical inference using Alexa as a historical precedent, showing how edge and cloud complement each other. |
Statements from this episode (15)
AWS's Sivasubramanian: AI scaling is not close to hitting a wall
“One, I don't think we are anywhere close to yet hitting that wall yet. So I do think we are going to actually see a lot of net new innovations when it comes to optimizing how to actually train these models in parallel and get better Utilization so that we can …”
Sivasubramanian: Trainium2 offers 4x faster training and 2x energy efficiency
“AWS actually, for instance, we have invested in technologies such as Trinium-II chips, which actually can deliver up to four X faster training. And, ah, also it is while improving energy efficiency up to two times, as an example.”
AWS AI VP: New model architectures will reshape AI scaling laws
“And if you see there are starting to see new kinds of architectures that are going to change how these models scale in the future. And I suspect, ah, that is going to change, ah, how These scaling laws are going to be reshaped as well when it comes to that is …”
AWS AI Chief: AI architectures beyond pure transformers are absolutely necessary
“I actually think, ah, new architectures are absolutely necessary in the future, ah, and, ah, there are already hybrid architectures evolving. I mean, ah, I mean in terms of state space models to actually connectors hybrid models between state space to transfor…”
AWS AI Chief: Enterprise AI needs smaller distilled models, not monolithic LLMs
“To put these LLMs to work to solve real world problems, you got to actually take some of these models and then customize it, and the end result is not the biggest model. It is actually a much more customized, smaller model or a distal model to solve specific b…”
Kantrowitz: Training on enterprise data will not advance the AI field
“If you start changing with trading with Rocket Mortgage data, you might just get a better application for Rocket Mortgage. You're not gonna advance the field of AI by getting mortgage data in there.”
AWS's Sivasubramanian: No Single AI Model Will Rule the World
“And that's why I keep saying no one model and will rule the world in the future.”
Sivasubramanian: Anthropic will use AWS Trainium and Inferentia for future models
“Anthropic will also use AWS Tranium and Inferentia tips to build, train, and deploy their future models and as well.”
AWS AI Chief: Inferentia chips can cut AI computing costs by 40%
“So the EC two instances that are powered by our Inferentia chips deliver up to 50% better price performance per watt over comparable EC-II as well, and can reduce cost by up to 40% as well.”
AWS VP: Pure LLMs Won't Deliver Effective AI Reasoning Without Other Tools
“Reasoning capability is not, if we view it as purely like everything is going to be LLM driven, we as industry are going to be very, very not satisfied. That's why it has to be actually more iterative. And we work with LLM as one of the ingredients. It is not …”
AWS VP: LLM architectures will shift to hierarchical edge-and-cloud inference
“So, I expect LLNs to evolve in the same way, where there are a lot of, ah, decisions, especially if you have a powerful computing device, ah, which, ah, many smartphones and others, ah, to be having that, You can actually run some of those, ah, simple LLM thin…”
AWS VP: GenAI apps will chain multiple models and data connectors
“The way these generative AI applications are going to be built is actually like series of things chained together. Some of it is model inference. Some of it is actually workflows and whatnot. And that's why that pattern is going to be important. And they are g…”
Sivasubramanian: Amazon Bedrock guardrails improve AI model accuracy up to 84%
“We see up to 40, 84 percent improvement in further accuracy because of those capability as well.”