Bernhardsson: Modal can typically scale customers to 1,000 GPUs within minutes
Erik Bernhardsson · Can AI Infrastructure Work Like Magic? Erik Bernhardsson, CEO, Modal · Oct 31, 2024 · at 20:12
Erik Bernhardsson is the CEO of Modal, a serverless cloud execution platform for data and AI workloads.
0:00 / 0:07exact quote · 7.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And then if you one day need a thousand GPUs, we can get you, we can typically get you a thousand GPUs like pretty quickly, like talking minutes.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Erik Bernhardsson
Bernhardsson: GPU alt-clouds are real estate businesses adding little stack value
“It's like sort of like almost like real estate, right? Like they own sort of, you know, real estate and they rent it out, arguably sort of, you know, we work, you could say, if you want to be, you know, a little cynical, but like not a ton of value, like highe…”
Bernhardsson: LLM inference compute is shifting toward specialized chips like crypto mining
“I do think it's kind of fascinating how it's like kind of going in the same direction as like crypto mining towards more like specialized hardware.”
Bernhardsson: Modal discarded Kubernetes and Docker to build a custom stack
“So we built our own, we threw out Kubernetes out the window. We threw out Docker out the window. We built our own File system in order to optimize for how container images are distributed. We built our own scheduler in order to maintain this pool of workers an…”
Bernhardsson: AI inference spend will dominate model training spend long-term
“It's obvious that like in the long run, like inference spend will dominate just for like the obvious reason that like, that's where you make the money, right? Like, you know, when you're training a big model, that's like, you know, you, you're spending a lot o…”
Bernhardsson: Serverless architecture fits AI workloads better than web backends
“So it's sort of always felt to me like, you know, in, in hindsight, it looks like the right idea, but applied to the wrong problems. And I always like wonder, you know, now, like I'm looking at like what we built at modal. Like I I'm kind of convinced that lik…”
Bernhardsson: Demands for self-hosting mean the product isn't compelling enough
“One trap that I think a lot of startups fall into is that, you know, you go out and talk to a lot of customers and they sort of insist on, on self-hosting. In my opinion, like, you know, sometimes that's actually like a true, like hard concerns, but in many ca…”