Jun 13, 2025 · 1h 18m · latent-space
The Shape of Compute (Chris Lattner of Modular)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this in-depth discussion, Modular founder and CEO Chris Lattner joins Alessio Fanelli and Shawn 'swyx' Wang to explore the architecture of Mojo and MAX, the shifting economics of AI inference, and the engineering principles required to build portable, high-performance compute infrastructure.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 13.4% of the talking time here. How this is scored →
The hosts as informed peer 4.5 Guest teaching 5.9 Guest disagreement 2.4 The hosts pushing back 1.6
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 9:00 Chris dismisses the narrative that replacing CUDA is impossible
Chris forcefully rejects the conventional tech wisdom that a startup cannot displace CUDA, citing his career track record overcoming identical skepticism with LLVM, Swift, and MLIR.
Hardest push from the hosts ▶ 8:57 Swyx presses Chris on the definition of impossibleSwyx directly challenges Chris's framing of industry critics, pressing him to clarify whether commentators literally meant impossible or simply very difficult.
Biggest teaching moment ▶ 38:20 Chris breaks down the economics of GPU vs CPU cloud infrastructureChris educates the hosts on the structural difference between stateless elastic CPU hosting and stateful GPU capacity planning where hardware depreciates before long-term commits end.
The host holds their own ▶ 1:00:45 Swyx links chain-of-thought inference directly into training loopsSwyx demonstrates technical command by pointing out that modern test-time compute and RL make inference an essential phase of model training, validating Modular's architectural focus.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Modular's Three-Year Evolution from R&D to Product Execution | 3 | 6 | 2 | 1 | Swyx and Alessio set up the context around Modular's progress. Chris provides a deep dive into Modular's first three stealth R&D years, explaining why they refused to use CUDA and set a high bar of beating NVIDIA on LLaMA 3 serving before publicly launching. | |
| Engineering Roadmap: From CPU Compilers to GPU LLM Serving | 4 | 7 | 4 | 3 | Alessio asks about moving from CPU compilers to GPUs. When Swyx questions whether industry naysayers meant 'impossible or very hard', Chris details how conventional wisdom repeatedly declared LLVM, Swift, and MLIR impossible before their adoption. | |
| Architectural Breakdown: The MAX Engine and Mojo Concentric Circles | 5 | 7 | 3 | 2 | Alessio prompts Chris to walk through the architecture of the MAX engine. Chris details the concentric circle model starting from Mojo, offering a sharp critique of C++ complexity and explaining binding-free Python acceleration. | |
| Inference Engines and the Power of Modular Software Architecture | 5 | 6 | 3 | 2 | Swyx asks about community rivalries between vLLM and SGLang and whether modular design entails architectural trade-offs. Chris explains how monolithic frameworks become disposable while composable systems survive hardware shifts. | |
| Democratizing Inference and Fostering Ecosystem Contributions | 4 | 6 | 2 | 1 | Swyx notes the democratization theme in Modular's mission. Chris reframes the history, noting that PyTorch successfully democratized training in 2017 but inference remained an opaque black art reserved for big tech labs. | |
| Modular's Business Strategy and Cloud GPU Economics | 4 | 7 | 2 | 1 | Swyx inquires about Modular's commercial monetization model versus cloud inference endpoints. Chris breaks down the economic divergence between stateless CPU cloud scaling and stateful GPU reserved commitments facing rapid hardware obsolescence. | |
| Lessons in Language Launching: Comparing Swift and Mojo | 4 | 6 | 3 | 2 | Swyx asks for a comparison between rolling out Swift at Apple and launching Mojo. Chris politely restates the prompt into a lessons-learned framing, contrasting Swift's secret 1.0 NDA launch with Mojo's transparent 0.1 dogfooding. | |
| Big Tech AI Dynamics, Google's Legacy, and the DeepSeek Catalyst | 6 | 7 | 3 | 2 | Alessio and Swyx guide the conversation to Google's AI history and DeepSeek's open source shockwave. Chris explains how DeepSeek reverse engineering PTX tensor core instructions mirrors standard practices of top compiler engineers and reveals why Blackwell kernel rewrites are needed. | |
| Reasoning Models, Reinforcement Learning, and the Centrality of Inference | 7 | 4 | 1 | 2 | Swyx demonstrates strong domain knowledge by noting that reasoning models and reinforcement learning collapse the distinction between training and inference. Chris affirms the insight, connecting it back to early DeepMind AlphaGo infrastructure challenges. | |
| Founder Leadership, Team Scaling, and Daily Productivity Routines | 3 | 4 | 1 | 1 | Alessio asks Chris about his founder routine and managing startup team dynamics. Chris reflects candidly on personal emotional adjustments when running a startup versus scaling teams in large tech corporations. | |
| AI Coding Agents, The Role of Human Code, and Research Tracking | 4 | 5 | 2 | 1 | Swyx asks how AI coding assistants influence language adoption and compiler writing. Chris argues that programming languages remain fundamental because code exists primarily for human reasoning and constraint management rather than machine execution alone. |
Statements from this episode (23)
Alternative AI stacks must match state-of-the-art performance to be useful
“Building An alternate AI stack that is not as good as the existing one isn't very useful because people will always evaluate it against state of the art.”
Modular will soon launch support for AMD MI300 and MI325 GPUs
“We're about to launch our AMD and my 303 25 support. That'll be a big deal for the industry.”
First-time Mojo developers built a GPU training system in one day
“The winning team for the hackathon took that four person team. And one day they had not used Mojo before they hadn't programmed GPUs before. And they had built a training system. They wrote an atom optimizer, a bunch of training kernels. They built a simple ba…”
Modular beat Intel MKL matrix multiplication speed in its first year
“First year was prove compilation philosophy. And so this was writing very abstract compiler stuff. And then prove that we can make a matrix multiplication go faster than Intel MKL's matrix multiplication on Intel Silicon and make it configurable and multiple D…”
Lattner: Modular MAX is 'more open source' than vLLM
“This thing's more open source than VLM because VLM depends on all these crazy binary CUDA kernels and stuff like this that are just opaque blobs from NVIDIA, right?”
Lattner says C++ 'sucks' for modern accelerated compute
“Let me be the first to tell you, and I can say this now, I feel comfortable saying this, that C++ sucks.”
Lattner says Mojo is 10,000x faster than Python and beats Rust
“Mojo is not just a little faster than Python, it's faster than Rust. So it's like tens of thousands of times faster than Python, and it's in the Python family.”
Lattner calls vLLM a 'hot mess' due to too many stakeholders
“VLM seems much more like a massive community with a lot of stakeholders, a lot of stuff going on, and it's kind of a hot mess.”
Modular's Mojo FlashAttention beats Tri Dao's reference implementation
“We're beating the tree DAO reference implementation that everybody uses, for example, right? Written fully in Mojo. Again, all of our, GPU kernels are written in Mojo. You can go see the history of the team building this, and it was done in just a few weeks, r…”
Modular's team has grown to over 100 employees
“So we're a small team. I mean, we're over a hundred people, but we're compared to the size of the problem we're taking on.”
PyTorch democratized model training, but AI inference remains undemocratized
“And so things like PyTorch came on the scene and I think PyTorch gets all credit for democratizing model training, right? It's taught to pretty much every computer science student that graduates. That's a huge deal, but nobody democratized inference. Inference…”
Intelligent compute routing delivers 5x performance TCO benefit
“You can get like five X performance TCO benefit by doing intelligent routing.”
Modular's Mojo and MAX are free on NVIDIA and CPUs
“The max framework and the mojo language, free to use on NVIDIA and CPUs, any scale, go nuts, do whatever you want.”
Modular monetizes cluster management and support on a per-GPU basis
“If you want cluster management, and you want enterprise support, and you want things like this, then you can pay on a per GPU basis, and you can contact our sales team, and then we can work out a deal, and we can work with you directly.”
An industry-leading state-of-the-art AI model may launch on MAX soon
“We may have an industry leading state of the art model launching first on Mac soon.”
Only 250 people knew about Apple's Swift language before its 2014 launch
“Started in 2010, it launched publicly by Apple in 2014, ok? And by the time it launched in 2014, only about 250 people in the world knew about it. Most of whom were in my team, about 200 and something of them were in my team, and then it was senior execs, mark…”
Modular has open-sourced 650,000 lines of Mojo code
“Modular is Mojo's first customer. Like we have more Mojo code in our repository than any other language. And it's open source. Like, and we open source like 650,000 lines of Mojo code.”
NVIDIA Blackwell is not backward-compatible with Hopper kernels
“Well, so it's not widely known, but Blackwell is not compatible with Hopper. Hopper kernels don't always run on Blackwell, for example.”
Modular completely eliminates and replaces NVIDIA's CUDA stack
“In the case of Modular, we go literally, like, we only work at that level, because we get rid of all of CUDA, right? And so we've replaced the entire stack, and so we only do that.”
DeepSeek's open releases pulled global AI progress forward by six months
“But I think that what it did is it pulled forward progress in AI by like six months.”
AI training scales with research teams; inference scales with customer base
“Training scales the size of your research team. Inference scales the size of your customer base.”
CPU latency in KV cache management bottlenecks GPU utilization
“Like your eviction policy runs on a CPU. Like that radix hashing algorithm and block hashing and all that stuff happens like primarily CPU. That's really important for performance because if you have latency in these steps, like you're not keeping your GPU uti…”
AI coding assistants excel at boilerplate, not inventing new algorithms
“People within the company are dabbling with cloud code and some of the stuff, and so I don't have personal experience with that, but for me, I found that it's mostly, it is very useful, but it's really about boilerplate, and so it's not about inventing new alg…”