Feb 19, 2024 · 1h 8m · latent-space
Truly Serverless Infra for AI Engineers - with Erik Bernhardsson of Modal
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space Podcast, Modal founder and CEO Erik Bernhardsson discusses modernizing cloud infrastructure for AI and data engineering, highlighting custom container runtimes, serverless GPU economics, and the developer experience of self-provisioning Python systems.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 28.3% of the talking time here. How this is scored →
The hosts as informed peer 5.8 Guest teaching 5.8 Guest disagreement 2.0 The hosts pushing back 1.5
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 59:42 Erik mocks extended hardware depreciation schedules
When Swyx brings up CTOs who claim they can make GPUs last seven years to justify buying hardware, Erik aggressively refutes the framing by comparing it to the Waste Management accounting fraud scandal where overextended truck depreciation led to restatements.
Hardest push from the hosts ▶ 28:44 Swyx rejects Modal's data teams positioningSwyx openly challenges Erik's persistent marketing of Modal as built for data teams, arguing that AI engineers represent the actual user base and citing empirical proof from the Vercel AI Accelerator.
Biggest teaching moment ▶ 11:54 Erik explains low-level container runtime bypassesErik explains step-by-step why Docker image pulls are wasteful and details how building a custom network virtual file system hooked directly to runC allows instant container cold starts without downloading multi-gigabyte layers.
The host holds their own ▶ 16:47 Swyx explains the necessity of self-provisioning runtimesSwyx lays out his thesis on self-provisioning runtimes, contrasting the awkward compile steps of AWS CDK and CloudFormation with true infrastructure-application convergence and programming language ergonomics.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Early Career at Spotify, Luigi, and Vector Databases | 6 | 5 | 1 | 1 | Swyx demonstrates familiarity with Bernhardsson's early career at Spotify, his open-source work on Luigi and Annoy, and vector indexing algorithms like HNSW. Bernhardsson explains the early pre-hype landscape of vector databases in 2012 and how Annoy used recursive hyperspace splitting. | |
| Scaling Better.com and Conceptualizing Modal's Compute Vision | 5 | 6 | 2 | 1 | Erik details why existing cloud infrastructure and Kubernetes failed data teams due to burstiness, GPU management, and fragile dependency environments. Swyx interjects with domain context on the modern data stack and feedback loops. | |
| Re-Engineering Container Runtimes and Custom Network File Systems | 4 | 8 | 1 | 0 | Erik educates the hosts on how Modal achieved sub-second container cold starts by bypassing standard Docker image pull mechanisms, using low-level runC primitives, and inventing a custom network-based virtual file system with deduplicated page caching. | |
| Self-Provisioning Runtimes and Python Developer Experience | 8 | 4 | 1 | 2 | Swyx demonstrates deep domain expertise regarding self-provisioning runtimes, contrasting AWS CDK and SAM with Modal's unified application-and-infrastructure code model. Erik enthusiastically acknowledges Swyx's framing and discusses the trade-offs of embedding infrastructure directly in Python decorators. | |
| Serverless AI Inference Dynamics and GPU Utilization | 6 | 7 | 2 | 1 | Alessio queries how serverless handles GPU utilization and batching compared to static memory allocation in foundational training. Erik explains how fast autoscaling and CPU/GPU memory checkpointing enable cost advantages despite higher per-hour GPU margins. | |
| Modal's Market Positioning: From Data Teams to Layer-Two Cloud | 7 | 5 | 3 | 5 | Swyx challenges Erik's insistence on labeling Modal's target user as data teams rather than AI engineers, citing adoption data from the Vercel AI Accelerator. Swyx also compares Modal to Replicate and Modular, prompting Erik to articulate Modal's layer-two cloud positioning. | |
| Overcoming the Cloud Graduation Problem and Enterprise Retention | 5 | 6 | 2 | 1 | Alessio asks about the enterprise graduation problem historically faced by PaaS providers like Heroku and how it applies to AI workloads. Erik explains why training lacks software differentiation compared to inference and how superior utilization economics retain customers. | |
| Language Model Workloads, Fine-Tuning, and Massive Parallelism | 6 | 5 | 1 | 1 | The hosts discuss LLM workloads including fine-tuning and self-contained agents like ErikBot. Swyx brings up his experience using Modal to orchestrate concurrency limits and rate-limiting when building Small Developer. | |
| Secure Code Sandboxes and Platform-for-Platforms Architecture | 6 | 6 | 1 | 1 | Swyx and Erik discuss secure code execution sandboxes built on gVisor. Swyx connects this architectural evolution to platform-for-platforms patterns, referencing Cloudflare's Functions-as-a-Service model. | |
| AI Inference Price Wars and Value in Custom Software | 6 | 6 | 3 | 2 | Alessio highlights negative unit economics in commodity LLM inference price wars. Erik agrees that raw token serving has zero switching costs, comparing VC-subsidized inference to the late-90s fiber optics boom and defending Modal's focus on proprietary, custom pipelines. | |
| High-IO Roadmaps, Oracle Cloud GPUs, and Hardware Philosophy | 6 | 6 | 4 | 3 | Swyx brings up alternative perspectives on buying owned hardware and stretching depreciation schedules over seven years. Erik strongly rejects the premise for venture startups, citing the Waste Management accounting scandal where aggressive truck depreciation forced massive earnings restatements. | |
| Competitive Programming Culture and Advice for Deep Tech Founders | 4 | 6 | 3 | 0 | Alessio and Swyx explore Erik's background in competitive programming (IOI) and high engineering talent density at Modal. Erik delivers a passionate closing thesis criticizing startups that act as thin Kubernetes wrappers instead of diving deep into systems infrastructure. |
Statements from this episode (31)
Daniel Ek Is a Generational CEO Among Swedish Startups
“Overall, I mean, I think he was a great CEO, like, definitely, you know, up there, like, generational CEO, at least for, like, Swedish startups.”
Better.com Grew to 10,000 Employees Before Shrinking Back to 1,000
“Yeah, I mean, the company, like, grew from, like, 10 people when I joined to 10,000, now it's back to a thousand, and, but yeah, they actually went public a few months ago, kind of crazy.”
Developer Productivity Is Best Measured by Iteration Feedback Speed
“I think the best way to measure developer productivity is, like, in terms of the feedback loops. Like, it, like, in, like, How quickly when you iterate, like when you write code, like how quickly can you get feedback? And at the innermost loop, it's like runni…”
Kubernetes Works for Backend Teams but Fails Data Teams
“Like in particular, like Kubernetes, I feel like it's like kind of worked okay for backend teams, but not so well for data teams.”
99% of Data in Multi-Gigabyte Docker Images Is Never Read
“It turns out, like, when you start a Docker container, like, first of all, like, most Docker images are, like, several gigabytes, and, like, 99% of that is never going to be consumed.”
Modal's Custom Container File System Enables Sub-Second Cloud Launches
“So that was, like, the first, like, sort of stuff we started working on was, like, let's build this, like, container file system, And, you know, coupled with like, you know, just using RunC directly. And that actually enabled us to like get to this point of li…”
AWS Has Always Struggled With Developer Experience
“AWS has always like struggled with developer experience”
Developer Productivity Will Continue Growing 10x Per Decade
“You look at the developers are, I've been getting like probably 10 X more productive every decade for the last four, four decades or something that was kind of crazy. Like on an exponential scale, we're talking about 10 X or is there a 10,000 X like, you know,…”
Serverless Architecture Fits Generative AI Far Better Than Backend Services
“Serverless always, to me, felt like a little bit of, like, a solution looking for a problem. I don't know, I don't actually, like, don't think, like, backend is, like, the problem that needs serverless, or, like, not as much, but I look at data, and in particu…”
Modal Had Little Traction Until the Release of Stable Diffusion
“That's, like, you know, a year and a half in, like, we barely had any users or any revenue, and, like, we were, like, well, maybe we should look at, like, some use case, trying to think of use cases, and that was around the time, same, same time Stable Edition…”
Modal Users Save Money on Inference Despite Higher GPU Rates
“In many cases, like users who do inference on those platforms or those clouds even though we charge a slightly higher price per, per GPU hour, a lot of users like moving their large scale inference use cases to model, like end up saving a lot of money because …”
Modal Implemented Full CPU Memory Checkpointing for Container Restores
“We've implemented recently CPU memory checkpointing, so we can take running containers and snapshot the entire CPU, like including registers and everything, and restore it from that point, which means we can restore it from a like initialized state.”
Modal Is Not Designed for Training Large Foundation Models
“When you get to like training, like very large foundational models, that that's a use case we don't support super well because that's very high IO, you know, you need to have like infinite band and all these things. And those are things we haven't supported ye…”
Modal Was the Top Tool Used in Vercel's AI Accelerator
“And in the Versailles AI accelerator, a bunch of startups gave like free credits and like signups and talks and all that stuff. The only ones that stuck are the people, are the ones that actually appealed to engineers and the top usage, the top tool used by fa…”
Suno Uses Modal for Its AI Music Production Infrastructure
“Yeah, so, I mean, they're using model for, like, production infrastructure, like, they have their own, like, custom model, like, custom code and custom weights, you know, for AI-generated music, Suno.ai.”
VCs Will Eventually Force Modular to Offer a Hosted Cloud Service
“I'm sure their VCs at some point are gonna force them to reconsider.”
Snowflake Proved Layer-Two Clouds Can Win on Developer Experience
“And I think Snowflake, you know, to me won because like, I mean, in the end, like AWS makes all the money anyway, like, and like Snowflake just had the ability to like focus on like developer experience or like, you know, or user experience. And to me, like re…”
Cloud Infra Startups Must Capture the Entire Hobbyist-to-Enterprise Spectrum
“The only way to do, to build infrastructure to build a successful infrastructure company in the long run in, in, in the cloud today is, You have to appeal to the entire spectrum, right? Or at least like the enterprise, like you have to capture the enterprise m…”
Buying Dedicated GPU Clusters Offers Better Economics for Model Training
“In training, I think, you know, there's less software differentiation. So in training, I think there's certainly, like, better economics of, like, buying big clusters.”
Only Tech Giants Will Build In-House AI Infrastructure Long-Term
“I think a lot of these companies over in the long run, like, you know, they're accepted maybe super big ones, like, you know, the Facebook and Google, they're always gonna build their own ones, but like everyone else, like some extent, you know, I think they'r…”
Modal Will Not Try to Compete Directly With OpenAI's API
“So many people get started with APIs and that's just, you know, they're just dominating a space in particular, open AI. Right. And that's not necessarily like a place where we aim to compete. I mean, maybe at some point, but like, it's just not like a core foc…”
Modal Can Fan Out Workloads to Thousands of GPUs in Minutes
“It's like pretty easy in Moto, like, fan out to like, you know, at least like a hundred GPUs, like in a few seconds, and you know, if you give it like a couple of minutes, like we can, you know, you can fan out to like thousands of GPUs.”
Modal's Sandbox Feature Is Evolving Into a Platform for Platforms
“The core, like, container infrastructure we offer could actually be, like, you know, unbundled from, like, the client SDK and offered to, like, other, you know, like, we're talking to a couple of, like other companies that want to run, you know, through their,…”
Switching Costs for Open-Source LLM Inference Providers Are Zero
“The LLM space is, like, the opposite. Like, the switching cost of LLMs is zero, right? Like, if all you're doing is, like, straight up, like, at least, like, open source, right? Like, if all you're doing is, like, you know, using some, you know, inference endp…”
VCs Subsidizing AI Inference Will Not See Their Expected Returns
“In the end, like, I don't think VCs will have the return they expected. Like, you know, in these things, but guess who's going to benefit? Like, you know, it's the consumers, right? Like someone's like reaping that the value of this. And that's, I think an ama…”
AI Inference Margins Will Rise While Consumer Prices Stay Low
“Margins are gonna go up for sure, but I don't know if prices will go up, because like, GPU prices have to drop eventually, right? So like, you know, like in the long run, I still think like prices may not go up that much but certainly margins will go up.”
Oracle Cloud GPUs Actually Offer Great Value for Money
“I love Oracle's GPUs. I mean, I don't, like, I don't know why, you know, what the economics looks like for Oracle, but like, I think they're great value for money. Like, we run a bunch of stuff in Oracle and They have bare metal machines with, like, two teraby…”
Buying Physical Hardware Is a Poor Use of Venture Capital
“My feeling is that when you're a venture-funded startup, like, buying physical hardware is maybe not the best use of the money.”
AI Productivity Gains Will Ultimately Increase Demand for Software Engineers
“Like, I think we can, you know, 10 x the amount of developers, and still, you know, have a lot of people making a lot of money, you know, building amazing software, and also being, while at the same time being more productive. Like, I never understood this, li…”
Modal Uses Mixed Integer Programming to Optimize Cloud Resource Allocation
“You know, the resource allocation, like turns out like that actually, like you can phrase that as a mixed integer programming problem. Like we now have that running in production, like constantly optimizing how we allocate cloud research.”
Startups Must Build Deep Infrastructure Rather Than Thin Kubernetes Wrappers
“And so one of my frustrations has been like so many startups are like, in my opinion, like Kubernetes wrappers and like, you know, like, and not very like thick wrappers, like fairly thin wrappers. And I think, you know, every startup is a wrapper to some exte…”