Bernhardsson: Serverless architecture fits AI workloads better than web backends
Erik Bernhardsson · Can AI Infrastructure Work Like Magic? Erik Bernhardsson, CEO, Modal · Oct 31, 2024 · at 28:24
Erik Bernhardsson, founder and CEO of Modal, explains why serverless computing struggled to win over traditional backend web developers but fits AI and data engineering teams perfectly.
0:00 / 0:28exact quote · 28.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So it's sort of always felt to me like, you know, in, in hindsight, it looks like the right idea, but applied to the wrong problems. And I always like wonder, you know, now, like I'm looking at like what we built at modal. Like I I'm kind of convinced that like serverless is actually makes a lot more sense for like data AI machine learning applications for a lot of different reasons, but you know, like you have like very bursty things. You have a lot of different containers. You don't necessarily care about like hyper low latency, I think there's a lot of different reasons why it's actually, like, makes a lot of sense for those types of use cases.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Erik Bernhardsson
Bernhardsson: GPU alt-clouds are real estate businesses adding little stack value
“It's like sort of like almost like real estate, right? Like they own sort of, you know, real estate and they rent it out, arguably sort of, you know, we work, you could say, if you want to be, you know, a little cynical, but like not a ton of value, like highe…”
Bernhardsson: LLM inference compute is shifting toward specialized chips like crypto mining
“I do think it's kind of fascinating how it's like kind of going in the same direction as like crypto mining towards more like specialized hardware.”
Bernhardsson: Modal discarded Kubernetes and Docker to build a custom stack
“So we built our own, we threw out Kubernetes out the window. We threw out Docker out the window. We built our own File system in order to optimize for how container images are distributed. We built our own scheduler in order to maintain this pool of workers an…”
Bernhardsson: AI inference spend will dominate model training spend long-term
“It's obvious that like in the long run, like inference spend will dominate just for like the obvious reason that like, that's where you make the money, right? Like, you know, when you're training a big model, that's like, you know, you, you're spending a lot o…”
Bernhardsson: Demands for self-hosting mean the product isn't compelling enough
“One trap that I think a lot of startups fall into is that, you know, you go out and talk to a lot of customers and they sort of insist on, on self-hosting. In my opinion, like, you know, sometimes that's actually like a true, like hard concerns, but in many ca…”
Bernhardsson: Lower GPU prices increase value of developer AI software
“If GPU prices were to crash, like the relative value of that software actually goes up. So like, I'm not necessarily sure that like a crash in GPU would be bad for us. I think it actually could be great for us because, you know, if GPU prices go down, there's …”