The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 46 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Conrad: GPU businesses succeed as pure real estate or pure software, not both
“The GPU clouds are fantastic real estate businesses. If you treat them like real estate businesses, you will make a lot of money. The, Cloud services you can make on that, all the software you want to make on that, you can do that fantastically. If you don't o…”
Startups Must Build Deep Infrastructure Rather Than Thin Kubernetes Wrappers
“And so one of my frustrations has been like so many startups are like, in my opinion, like Kubernetes wrappers and like, you know, like, and not very like thick wrappers, like fairly thin wrappers. And I think, you know, every startup is a wrapper to some exte…”
Bubna: Model APIs primarily serve a less sticky hobbyist market
“This is one thing we've kind of stayed away from is providing an API for models, because I think providing Model APIs is, some of it ends up serving like a really hobbyist market, which is much less sticky.”
Bubna: Kubernetes lacks burstiness support and has terrible developer experience
“Kubernetes is hard to manage. It's not built for burstiness and custom images and has a terrible developer experience.”
Bubna: Open-source DFlash matches proprietary speculative decoding performance
“Recently we shared our work on dflash, which is a block-based speculator, and we've open sourced all of it, so you can get, by using open source dflash, you can get the same performance as you would with one of the proprietary providers.”
Cloud Infra Startups Must Capture the Entire Hobbyist-to-Enterprise Spectrum
“The only way to do, to build infrastructure to build a successful infrastructure company in the long run in, in, in the cloud today is, You have to appeal to the entire spectrum, right? Or at least like the enterprise, like you have to capture the enterprise m…”
Modal Users Save Money on Inference Despite Higher GPU Rates
“In many cases, like users who do inference on those platforms or those clouds even though we charge a slightly higher price per, per GPU hour, a lot of users like moving their large scale inference use cases to model, like end up saving a lot of money because …”
Oracle Cloud GPUs Actually Offer Great Value for Money
“I love Oracle's GPUs. I mean, I don't, like, I don't know why, you know, what the economics looks like for Oracle, but like, I think they're great value for money. Like, we run a bunch of stuff in Oracle and They have bare metal machines with, like, two teraby…”
Modal Can Fan Out Workloads to Thousands of GPUs in Minutes
“It's like pretty easy in Moto, like, fan out to like, you know, at least like a hundred GPUs, like in a few seconds, and you know, if you give it like a couple of minutes, like we can, you know, you can fan out to like thousands of GPUs.”
Bubna: Modal's Biggest Use Case and Initial PMF Was Custom Non-LLM Inference
“Our biggest use case actually is elastic inference. And the thing we first found product market fit with was inference for custom models. So we kind of stayed away from the LM space and we were serving companies like Suno for audio, Runway for video, robotics …”
Modal CTO: Suno runs 100% of inference on Modal
“They use modal for all their inference, and that's because they have like a custom, they have completely custom model architecture, and that means that they have to be at the code level and tweak things that are not Yeah, it's an API.”
Bubna: AI Agent Developers Prefer TypeScript Over Python
“And the interesting thing with, like, the agent stuff is people use their TypeScript SDK a lot more because they're not actually doing anything that needs ML.”
Bubna: Modal is bullish on CI as coding agents run more tests
“We're very bullish and modal on the CI market as well because it has, There's more agents coding agents they're gonna run a lot more CI, and the preventives there can be much better.”
Bubna: Modal operates zero data centers, running entirely on NeoClouds
“We don't have our own data centers. We just run across a lot of Neo clouds.”
Bubna: Agent Experience Benefits From Typed Decorators Over Kubernetes YAML
“We think that the same benefits that apply for DX also actually apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and, like, write YAML that's not even typed when it can basically make a couple of changes in a decorat…”
Bubna: Modal uses eBPF-filtered TCP rather than VPN encryption for internal networking
“This is TCP, and we have eBPF programs that will reject or allow the TCP connection based on whether you're allowed to do it.”
Bubna: Modal built decorators to eliminate YAML and make infra dynamic
“That, that, that was really important because we really didn't want people to spend so much time writing YAML, and it seemed like you could really condense the surface area of what you're doing put it in code so you can actually operate on it, just like you ca…”
Bubna: Production AI agents require specialized sandboxes for compute and networking control
“You need a lot more control over your compute primitive on things like what sort of, how do you persist different files that the agent has access to, and how do you snapshot and restore? How do you control the networking? Maybe you want GPUs. When you get to t…”
Bubna: Batch compute demand comes mainly from non-LLM workloads like computational biology
“The demand that we see for something like that is actually not for LLMs. Although sometimes people want to run evals and do synthetic data prep and there it makes sense. But it's from a lot of non LLM companies like people who are doing computational bio, like…”
Bubna: AI agents struggle to reason through logs and observability
“I think the things that sometimes agents struggle with without right guidance and a skill is how to use the rest of our observability. Like, how to, something is failing, like, How do you look at the logs and then update the right thing? It's sort of reasoning…”
Bubna: Modal's custom reliability layer insulates users from GPU hardware drops
“That's why it's something we've invested a lot of time in is actually building our own reliability layer on top. So if the GPU falls off the bus or something happens, we user workloads are not affected.”
Swyx: Suno grew from zero to $20M ARR running on Modal
“Suno ramp has rated as one of the top ranked fastest growing startups of the year. I think the last public number is like zero to twenty million this year in ARR and Suno runs on Moto. So Suno itself is not GPU rich, but they're just doing the training on, on …”
Suno Uses Modal for Its AI Music Production Infrastructure
“Yeah, so, I mean, they're using model for, like, production infrastructure, like, they have their own, like, custom model, like, custom code and custom weights, you know, for AI-generated music, Suno.ai.”
Modal Was the Top Tool Used in Vercel's AI Accelerator
“And in the Versailles AI accelerator, a bunch of startups gave like free credits and like signups and talks and all that stuff. The only ones that stuck are the people, are the ones that actually appealed to engineers and the top usage, the top tool used by fa…”
Switching Costs for Open-Source LLM Inference Providers Are Zero
“The LLM space is, like, the opposite. Like, the switching cost of LLMs is zero, right? Like, if all you're doing is, like, straight up, like, at least, like, open source, right? Like, if all you're doing is, like, you know, using some, you know, inference endp…”
Modal's Sandbox Feature Is Evolving Into a Platform for Platforms
“The core, like, container infrastructure we offer could actually be, like, you know, unbundled from, like, the client SDK and offered to, like, other, you know, like, we're talking to a couple of, like other companies that want to run, you know, through their,…”
Modal Will Not Try to Compete Directly With OpenAI's API
“So many people get started with APIs and that's just, you know, they're just dominating a space in particular, open AI. Right. And that's not necessarily like a place where we aim to compete. I mean, maybe at some point, but like, it's just not like a core foc…”
Bubna: Modal provides approximately 3 Tbps internal networking bandwidth
“And we have I think like three terabit per second internal networking, which is the standard that's needed.”
Bubna: Modal built agent sandboxes and published smol-developer recursive loops early
“We built sandboxes in May of the year before anyone knew this was going to be a thing. And the first example we published was we took a small developer and put in a loop so the agent can iterate on itself.”
Bubna: Modal is building a cheaper 24-hour batch compute tier
“One of the things we're building now is, like, a way for customers to get if they don't care about latency like, get much cheaper pricing and they'll get results back in, like, next 24 hours or something. Like a batch tier, essentially.”
Modal sandboxes support multi-container sidecar pods
“So actually, if you want Docker Compose our sandboxes now support this thing called Sidecarves. So you can, a sandbox is actually a pod of containers, and you can run multiple containers in a sandbox.”
Bubna: Modal Refocused Its SDK Team on Agent Experience
“We've actually changed our SDK team to think about agent experience, sort of developer experience”
Bubna: Modal's capacity pool spans 17 cloud providers
“We've built this capacity pool that spans 17 cloud providers. So we're very good at running on various kinds of cloud capacity across the world.”
Bubna: Ramp runs its external-facing accounting agent on Modal
“Ramp also runs their accounting agent on us, so their external facing agent.”
Bubna: Modal uses autonomous agents to run internal inference optimization sweeps
“Internal both training and inference teams actually use this sort of the general shape of this quite a bit. Like we have this one internal repo called auto inference, which essentially we've automated our own FDE efforts using this harness, which is the agent …”
Modal built an IPv6 overlay network for private container addressing
“We have this thing called I-SXPN, which we haven't talked about which is this, like, overlay network using IPv six addresses so if modal containers within the same workspace when this is enabled, can actually address each other using this private IPv six addre…”
Bubna: Modal Uses GPU Snapshotting to Speed Up Cold Starts
“We've incorporated GPU snapshotting to the product so we can actually take the GPU state, like your Torch compiler model, snapshot it, and the next call starts way faster.”
Bubna: Modal multi-node training targets post-training, not large-scale pre-training
“And we're not going for obviously like large scale pre-training runs. The thing that we've built multi-handle training for is we see a lot of smaller scale post-training like people are post-training like medium-sized fun models so they can get higher quality …”
Bubna: Ramp Inspect Succeeded Using Modal Snapshotting and Fast Scaling
“Ramp Inspect was a great example of a background agent that was really successful because they were able to use some of the primitives like snapshotting and fast scaling to just have something that feels really reactive and works well.”
Modal CTO: Production scale requires elastically scaling 1,000 to 1,500 GPUs quickly
“There it's not about scaling from zero to one, but it's how do we scale really elastically from, like, thousand to 1500 GPUs very quickly in, in a given region.”
Bubna: Modal embedded an engineer at Cognition to reduce communication latency
“We sent him over because the latency of communication was too high otherwise.”
Modal Implemented Full CPU Memory Checkpointing for Container Restores
“We've implemented recently CPU memory checkpointing, so we can take running containers and snapshot the entire CPU, like including registers and everything, and restore it from that point, which means we can restore it from a like initialized state.”
Modal Is Not Designed for Training Large Foundation Models
“When you get to like training, like very large foundational models, that that's a use case we don't support super well because that's very high IO, you know, you need to have like infinite band and all these things. And those are things we haven't supported ye…”
Modal Uses Mixed Integer Programming to Optimize Cloud Resource Allocation
“You know, the resource allocation, like turns out like that actually, like you can phrase that as a mixed integer programming problem. Like we now have that running in production, like constantly optimizing how we allocate cloud research.”
Modal Had Little Traction Until the Release of Stable Diffusion
“That's, like, you know, a year and a half in, like, we barely had any users or any revenue, and, like, we were, like, well, maybe we should look at, like, some use case, trying to think of use cases, and that was around the time, same, same time Stable Edition…”
Modal's Custom Container File System Enables Sub-Second Cloud Launches
“So that was, like, the first, like, sort of stuff we started working on was, like, let's build this, like, container file system, And, you know, coupled with like, you know, just using RunC directly. And that actually enabled us to like get to this point of li…”