Erik Bernhardsson: Very large-scale AI training is not a high priority for Modal
Erik Bernhardsson · Can AI Infrastructure Work Like Magic? Erik Bernhardsson, CEO, Modal · Oct 31, 2024 · at 21:51
Erik Bernhardsson, CEO of Modal Labs, discusses the infrastructure requirements for large-scale model training versus inference workloads.
0:00 / 0:12exact quote · 12.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It puts a lot of different, you know constraints on, on the infrastructure that we don't support today. It's something we're interested in, like, you know, somewhat like, you know, looking at down the road but it's not like a super high priority for the business right now.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Erik Bernhardsson
Bernhardsson: GPU alt-clouds are real estate businesses adding little stack value
“It's like sort of like almost like real estate, right? Like they own sort of, you know, real estate and they rent it out, arguably sort of, you know, we work, you could say, if you want to be, you know, a little cynical, but like not a ton of value, like highe…”
Bernhardsson: LLM inference compute is shifting toward specialized chips like crypto mining
“I do think it's kind of fascinating how it's like kind of going in the same direction as like crypto mining towards more like specialized hardware.”
Bernhardsson: Modal discarded Kubernetes and Docker to build a custom stack
“So we built our own, we threw out Kubernetes out the window. We threw out Docker out the window. We built our own File system in order to optimize for how container images are distributed. We built our own scheduler in order to maintain this pool of workers an…”
Bernhardsson: AI inference spend will dominate model training spend long-term
“It's obvious that like in the long run, like inference spend will dominate just for like the obvious reason that like, that's where you make the money, right? Like, you know, when you're training a big model, that's like, you know, you, you're spending a lot o…”
Bernhardsson: Serverless architecture fits AI workloads better than web backends
“So it's sort of always felt to me like, you know, in, in hindsight, it looks like the right idea, but applied to the wrong problems. And I always like wonder, you know, now, like I'm looking at like what we built at modal. Like I I'm kind of convinced that lik…”
Bernhardsson: Demands for self-hosting mean the product isn't compelling enough
“One trap that I think a lot of startups fall into is that, you know, you go out and talk to a lot of customers and they sort of insist on, on self-hosting. In my opinion, like, you know, sometimes that's actually like a true, like hard concerns, but in many ca…”