Modal

“Yeah, so, I mean, they're using model for, like, production infrastructure, like, they have their own, like, custom model, like, custom code and custom weights, you know, for AI-generated music, Suno.ai.”

“And so one of my frustrations has been like so many startups are like, in my opinion, like Kubernetes wrappers and like, you know, like, and not very like thick wrappers, like fairly thin wrappers. And I think, you know, every startup is a wrapper to some exte…”

“And in the Versailles AI accelerator, a bunch of startups gave like free credits and like signups and talks and all that stuff. The only ones that stuck are the people, are the ones that actually appealed to engineers and the top usage, the top tool used by fa…”

“The LLM space is, like, the opposite. Like, the switching cost of LLMs is zero, right? Like, if all you're doing is, like, straight up, like, at least, like, open source, right? Like, if all you're doing is, like, you know, using some, you know, inference endp…”

“The core, like, container infrastructure we offer could actually be, like, you know, unbundled from, like, the client SDK and offered to, like, other, you know, like, we're talking to a couple of, like other companies that want to run, you know, through their,…”

“The only way to do, to build infrastructure to build a successful infrastructure company in the long run in, in, in the cloud today is, You have to appeal to the entire spectrum, right? Or at least like the enterprise, like you have to capture the enterprise m…”

“So many people get started with APIs and that's just, you know, they're just dominating a space in particular, open AI. Right. And that's not necessarily like a place where we aim to compete. I mean, maybe at some point, but like, it's just not like a core foc…”

“We've implemented recently CPU memory checkpointing, so we can take running containers and snapshot the entire CPU, like including registers and everything, and restore it from that point, which means we can restore it from a like initialized state.”

“When you get to like training, like very large foundational models, that that's a use case we don't support super well because that's very high IO, you know, you need to have like infinite band and all these things. And those are things we haven't supported ye…”

“You know, the resource allocation, like turns out like that actually, like you can phrase that as a mixed integer programming problem. Like we now have that running in production, like constantly optimizing how we allocate cloud research.”

“In many cases, like users who do inference on those platforms or those clouds even though we charge a slightly higher price per, per GPU hour, a lot of users like moving their large scale inference use cases to model, like end up saving a lot of money because …”

“I love Oracle's GPUs. I mean, I don't, like, I don't know why, you know, what the economics looks like for Oracle, but like, I think they're great value for money. Like, we run a bunch of stuff in Oracle and They have bare metal machines with, like, two teraby…”

“That's, like, you know, a year and a half in, like, we barely had any users or any revenue, and, like, we were, like, well, maybe we should look at, like, some use case, trying to think of use cases, and that was around the time, same, same time Stable Edition…”

“It's like pretty easy in Moto, like, fan out to like, you know, at least like a hundred GPUs, like in a few seconds, and you know, if you give it like a couple of minutes, like we can, you know, you can fan out to like thousands of GPUs.”

“So that was, like, the first, like, sort of stuff we started working on was, like, let's build this, like, container file system, And, you know, coupled with like, you know, just using RunC directly. And that actually enabled us to like get to this point of li…”

“Suno ramp has rated as one of the top ranked fastest growing startups of the year. I think the last public number is like zero to twenty million this year in ARR and Suno runs on Moto. So Suno itself is not GPU rich, but they're just doing the training on, on …”

“The GPU clouds are fantastic real estate businesses. If you treat them like real estate businesses, you will make a lot of money. The, Cloud services you can make on that, all the software you want to make on that, you can do that fantastically. If you don't o…”

“Our biggest use case actually is elastic inference. And the thing we first found product market fit with was inference for custom models. So we kind of stayed away from the LM space and we were serving companies like Suno for audio, Runway for video, robotics …”

“And we have I think like three terabit per second internal networking, which is the standard that's needed.”

“This is one thing we've kind of stayed away from is providing an API for models, because I think providing Model APIs is, some of it ends up serving like a really hobbyist market, which is much less sticky.”

“They use modal for all their inference, and that's because they have like a custom, they have completely custom model architecture, and that means that they have to be at the code level and tweak things that are not Yeah, it's an API.”

“We built sandboxes in May of the year before anyone knew this was going to be a thing. And the first example we published was we took a small developer and put in a loop so the agent can iterate on itself.”

“One of the things we're building now is, like, a way for customers to get if they don't care about latency like, get much cheaper pricing and they'll get results back in, like, next 24 hours or something. Like a batch tier, essentially.”

“And the interesting thing with, like, the agent stuff is people use their TypeScript SDK a lot more because they're not actually doing anything that needs ML.”

“We're very bullish and modal on the CI market as well because it has, There's more agents coding agents they're gonna run a lot more CI, and the preventives there can be much better.”

“We don't have our own data centers. We just run across a lot of Neo clouds.”

“So actually, if you want Docker Compose our sandboxes now support this thing called Sidecarves. So you can, a sandbox is actually a pod of containers, and you can run multiple containers in a sandbox.”

“We think that the same benefits that apply for DX also actually apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and, like, write YAML that's not even typed when it can basically make a couple of changes in a decorat…”

“This is TCP, and we have eBPF programs that will reject or allow the TCP connection based on whether you're allowed to do it.”

“We've actually changed our SDK team to think about agent experience, sort of developer experience”

“That, that, that was really important because we really didn't want people to spend so much time writing YAML, and it seemed like you could really condense the surface area of what you're doing put it in code so you can actually operate on it, just like you ca…”

“Kubernetes is hard to manage. It's not built for burstiness and custom images and has a terrible developer experience.”

“We've built this capacity pool that spans 17 cloud providers. So we're very good at running on various kinds of cloud capacity across the world.”

“Ramp also runs their accounting agent on us, so their external facing agent.”

“You need a lot more control over your compute primitive on things like what sort of, how do you persist different files that the agent has access to, and how do you snapshot and restore? How do you control the networking? Maybe you want GPUs. When you get to t…”

“Internal both training and inference teams actually use this sort of the general shape of this quite a bit. Like we have this one internal repo called auto inference, which essentially we've automated our own FDE efforts using this harness, which is the agent …”

“The demand that we see for something like that is actually not for LLMs. Although sometimes people want to run evals and do synthetic data prep and there it makes sense. But it's from a lot of non LLM companies like people who are doing computational bio, like…”

“We have this thing called I-SXPN, which we haven't talked about which is this, like, overlay network using IPv six addresses so if modal containers within the same workspace when this is enabled, can actually address each other using this private IPv six addre…”

“We've incorporated GPU snapshotting to the product so we can actually take the GPU state, like your Torch compiler model, snapshot it, and the next call starts way faster.”

“And we're not going for obviously like large scale pre-training runs. The thing that we've built multi-handle training for is we see a lot of smaller scale post-training like people are post-training like medium-sized fun models so they can get higher quality …”

“Ramp Inspect was a great example of a background agent that was really successful because they were able to use some of the primitives like snapshotting and fast scaling to just have something that feels really reactive and works well.”

“Recently we shared our work on dflash, which is a block-based speculator, and we've open sourced all of it, so you can get, by using open source dflash, you can get the same performance as you would with one of the proprietary providers.”

“I think the things that sometimes agents struggle with without right guidance and a skill is how to use the rest of our observability. Like, how to, something is failing, like, How do you look at the logs and then update the right thing? It's sort of reasoning…”

“There it's not about scaling from zero to one, but it's how do we scale really elastically from, like, thousand to 1500 GPUs very quickly in, in a given region.”

“We sent him over because the latency of communication was too high otherwise.”

“That's why it's something we've invested a lot of time in is actually building our own reliability layer on top. So if the GPU falls off the bus or something happens, we user workloads are not affected.”