Modal

“What's shocking about those business models is they're growing almost as fast as the frontier labs in the early days. But burning very little cash.”

“Kubernetes is hard to manage. It's not built for burstiness and custom images and has a terrible developer experience.”

“That, that, that was really important because we really didn't want people to spend so much time writing YAML, and it seemed like you could really condense the surface area of what you're doing put it in code so you can actually operate on it, just like you ca…”

“We've actually changed our SDK team to think about agent experience, sort of developer experience”

“We think that the same benefits that apply for DX also actually apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and, like, write YAML that's not even typed when it can basically make a couple of changes in a decorat…”

“We sent him over because the latency of communication was too high otherwise.”

“We built sandboxes in May of the year before anyone knew this was going to be a thing. And the first example we published was we took a small developer and put in a loop so the agent can iterate on itself.”

“There it's not about scaling from zero to one, but it's how do we scale really elastically from, like, thousand to 1500 GPUs very quickly in, in a given region.”

“Our biggest use case actually is elastic inference. And the thing we first found product market fit with was inference for custom models. So we kind of stayed away from the LM space and we were serving companies like Suno for audio, Runway for video, robotics …”

“We've incorporated GPU snapshotting to the product so we can actually take the GPU state, like your Torch compiler model, snapshot it, and the next call starts way faster.”

“Recently we shared our work on dflash, which is a block-based speculator, and we've open sourced all of it, so you can get, by using open source dflash, you can get the same performance as you would with one of the proprietary providers.”

“Ramp Inspect was a great example of a background agent that was really successful because they were able to use some of the primitives like snapshotting and fast scaling to just have something that feels really reactive and works well.”

“We've built this capacity pool that spans 17 cloud providers. So we're very good at running on various kinds of cloud capacity across the world.”

“We don't have our own data centers. We just run across a lot of Neo clouds.”

“That's why it's something we've invested a lot of time in is actually building our own reliability layer on top. So if the GPU falls off the bus or something happens, we user workloads are not affected.”

“So actually, if you want Docker Compose our sandboxes now support this thing called Sidecarves. So you can, a sandbox is actually a pod of containers, and you can run multiple containers in a sandbox.”

“We have this thing called I-SXPN, which we haven't talked about which is this, like, overlay network using IPv six addresses so if modal containers within the same workspace when this is enabled, can actually address each other using this private IPv six addre…”

“This is TCP, and we have eBPF programs that will reject or allow the TCP connection based on whether you're allowed to do it.”

“And we have I think like three terabit per second internal networking, which is the standard that's needed.”

“And we're not going for obviously like large scale pre-training runs. The thing that we've built multi-handle training for is we see a lot of smaller scale post-training like people are post-training like medium-sized fun models so they can get higher quality …”

“Internal both training and inference teams actually use this sort of the general shape of this quite a bit. Like we have this one internal repo called auto inference, which essentially we've automated our own FDE efforts using this harness, which is the agent …”

“I think the things that sometimes agents struggle with without right guidance and a skill is how to use the rest of our observability. Like, how to, something is failing, like, How do you look at the logs and then update the right thing? It's sort of reasoning…”

“One of the things we're building now is, like, a way for customers to get if they don't care about latency like, get much cheaper pricing and they'll get results back in, like, next 24 hours or something. Like a batch tier, essentially.”

“The demand that we see for something like that is actually not for LLMs. Although sometimes people want to run evals and do synthetic data prep and there it makes sense. But it's from a lot of non LLM companies like people who are doing computational bio, like…”

“Ramp also runs their accounting agent on us, so their external facing agent.”

“You need a lot more control over your compute primitive on things like what sort of, how do you persist different files that the agent has access to, and how do you snapshot and restore? How do you control the networking? Maybe you want GPUs. When you get to t…”

“This is one thing we've kind of stayed away from is providing an API for models, because I think providing Model APIs is, some of it ends up serving like a really hobbyist market, which is much less sticky.”

“They use modal for all their inference, and that's because they have like a custom, they have completely custom model architecture, and that means that they have to be at the code level and tweak things that are not Yeah, it's an API.”

“We're very bullish and modal on the CI market as well because it has, There's more agents coding agents they're gonna run a lot more CI, and the preventives there can be much better.”

“And the interesting thing with, like, the agent stuff is people use their TypeScript SDK a lot more because they're not actually doing anything that needs ML.”

“The GPU clouds are fantastic real estate businesses. If you treat them like real estate businesses, you will make a lot of money. The, Cloud services you can make on that, all the software you want to make on that, you can do that fantastically. If you don't o…”

“We can't really use Docker and Kubernetes. So we're going to have to throw that out and probably going to have to build our own file system, which we did pretty early in a build our own scheduler and build our own container runtime. So that was like basically …”

“We run a very big compute pool, like thousands of GPUs and CPUs, and we make it very easy to get, you know, if you need a hundred GPUs, we can typically get you that within seconds.”

“The main thing that really started driving all the traction was when stable diffusion came out. And a bunch of people came to us and like, Hey, actually, this looks kind of cool. Like you have GPU access. It's very easy to, you know, you have to think about, y…”

“So, so one example of a customer I think is super cool building really amazing stuff is Suno. Which does AI generated music. So they run all their inference on modal very large scale.”

“Traditionally, most of modal has always been inference. Like that's been our main use case, but we're really interested also in training. So in particular, like probably focused more on these like shorter, like very bursty sort of experimental training runs, n…”

“I think there's so many benefits of this multi-tenant model in terms of capacity management that to me, it is very clearly like a big part of the future of AI is like running a big pool of compute and slicing it very dynamically.”

“I think eventually like for any company where model quality really matters, unless you kind of train your own model in the end, like, I feel like it's going to be hard to sort of defend the fact that like, You know, you're, you have a better solution. Cause li…”

“And that's actually a field where we start to see a lot more usage as model as well. Is there's a lot of, I feel like there's like a kind of a resurgence of computational biology.”

“Suno ramp has rated as one of the top ranked fastest growing startups of the year. I think the last public number is like zero to twenty million this year in ARR and Suno runs on Moto. So Suno itself is not GPU rich, but they're just doing the training on, on …”

“Modal makes it easy to build, scale and deploy applications in the data, AI, machine learning realm. So, so we basically, you can think of as like, we take, you write a little bit of Python code, and we take that code, we stick it in a container, we execute th…”

“I actually like them a lot. We use them. Big fan.”

“Model inference is probably the majority of our use case.”

“Like, so, I don't know, I'm a big believer that, like, you know, in the long run, like, people will train a lot of different custom models.”

“So we built our own, we threw out Kubernetes out the window. We threw out Docker out the window. We built our own File system in order to optimize for how container images are distributed. We built our own scheduler in order to maintain this pool of workers an…”

“We actually have a system that continuously looks at cloud pricing and solves an optimization problem to sort of, you know, figure out How do we allocate capacity across like, you know, a hundred different regions in order to deliver the capacity we need to ou…”

“And then if you one day need a thousand GPUs, we can get you, we can typically get you a thousand GPUs like pretty quickly, like talking minutes.”

“Suno which is AI generated music. And, you know, they have a big cluster. They train their own models outside of modal and then they use modal for the inference side.”

“It puts a lot of different, you know constraints on, on the infrastructure that we don't support today. It's something we're interested in, like, you know, somewhat like, you know, looking at down the road but it's not like a super high priority for the busine…”

“So it's sort of always felt to me like, you know, in, in hindsight, it looks like the right idea, but applied to the wrong problems. And I always like wonder, you know, now, like I'm looking at like what we built at modal. Like I I'm kind of convinced that lik…”

“In order to get to sort of latencies below, you know, a 152 hundred milliseconds, we probably need to split that up and run like a decentralized control plane.”

“Majority of the Lux AI portfolio is here. I think we have a pretty darn good portfolio but that means Hugging Face is headquartered here. Runway is headquartered here. Mosaic ML, which was acquired by Databricks, They, their research team is largely here. Osmo…”

“So that was, like, the first, like, sort of stuff we started working on was, like, let's build this, like, container file system, And, you know, coupled with like, you know, just using RunC directly. And that actually enabled us to like get to this point of li…”

“That's, like, you know, a year and a half in, like, we barely had any users or any revenue, and, like, we were, like, well, maybe we should look at, like, some use case, trying to think of use cases, and that was around the time, same, same time Stable Edition…”

“In many cases, like users who do inference on those platforms or those clouds even though we charge a slightly higher price per, per GPU hour, a lot of users like moving their large scale inference use cases to model, like end up saving a lot of money because …”

“We've implemented recently CPU memory checkpointing, so we can take running containers and snapshot the entire CPU, like including registers and everything, and restore it from that point, which means we can restore it from a like initialized state.”

“When you get to like training, like very large foundational models, that that's a use case we don't support super well because that's very high IO, you know, you need to have like infinite band and all these things. And those are things we haven't supported ye…”

“And in the Versailles AI accelerator, a bunch of startups gave like free credits and like signups and talks and all that stuff. The only ones that stuck are the people, are the ones that actually appealed to engineers and the top usage, the top tool used by fa…”

“Yeah, so, I mean, they're using model for, like, production infrastructure, like, they have their own, like, custom model, like, custom code and custom weights, you know, for AI-generated music, Suno.ai.”

“The only way to do, to build infrastructure to build a successful infrastructure company in the long run in, in, in the cloud today is, You have to appeal to the entire spectrum, right? Or at least like the enterprise, like you have to capture the enterprise m…”