The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 40 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 1 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Bubna: Speculative decoding accept length delivers multiplicative speedups over kernel tuning
“People talk a lot about, we made these kernels faster and whatnot, but improving kernel only give you like a few percentage points of improvement and increasing except length literally is a multiplicative decrease.”
Bubna: Kubernetes lacks burstiness support and has terrible developer experience
“Kubernetes is hard to manage. It's not built for burstiness and custom images and has a terrible developer experience.”
Bubna: System Observability Is Becoming More Important Than Reading Code
“You still need humans to go interpret what's going on and you know, make judgment calls and whatnot. And that's, I feel like maybe more important now than looking at the code itself.”
Bubna: Open-source DFlash matches proprietary speculative decoding performance
“Recently we shared our work on dflash, which is a block-based speculator, and we've open sourced all of it, so you can get, by using open source dflash, you can get the same performance as you would with one of the proprietary providers.”
Bubna: Sandboxes Require Hard Boundaries, Not LLM-Mediated Permissions
“I'm skeptical of LLM-mediated permissions for stuff that is At the sandbox level, because you do want hard boundaries. Otherwise, obviously someone can exfiltrate stuff.”
Bubna: Model APIs primarily serve a less sticky hobbyist market
“This is one thing we've kind of stayed away from is providing an API for models, because I think providing Model APIs is, some of it ends up serving like a really hobbyist market, which is much less sticky.”
Bubna: Modal built decorators to eliminate YAML and make infra dynamic
“That, that, that was really important because we really didn't want people to spend so much time writing YAML, and it seemed like you could really condense the surface area of what you're doing put it in code so you can actually operate on it, just like you ca…”
Bubna: Agent Experience Benefits From Typed Decorators Over Kubernetes YAML
“We think that the same benefits that apply for DX also actually apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and, like, write YAML that's not even typed when it can basically make a couple of changes in a decorat…”
Bubna: Modal's Biggest Use Case and Initial PMF Was Custom Non-LLM Inference
“Our biggest use case actually is elastic inference. And the thing we first found product market fit with was inference for custom models. So we kind of stayed away from the LM space and we were serving companies like Suno for audio, Runway for video, robotics …”
Bubna: RL Rollouts Are Extremely Bursty and Can Require 100,000 Sandboxes
“RL is insanely bursty. Like when you're doing rollouts you sometimes need a 100,000 sandboxes.”
Modal CTO: Production AI inference is difficult due to tail latency
“Running production grade inference is a hard and fair problem. Even if you subtract out the auto scaling, it's controlling things like tail latency and making sure every request is delivered at least once and whatnot.”
Bubna: Modal operates zero data centers, running entirely on NeoClouds
“We don't have our own data centers. We just run across a lot of Neo clouds.”
Bubna: Modal's custom reliability layer insulates users from GPU hardware drops
“That's why it's something we've invested a lot of time in is actually building our own reliability layer on top. So if the GPU falls off the bus or something happens, we user workloads are not affected.”
Bubna: Modal uses eBPF-filtered TCP rather than VPN encryption for internal networking
“This is TCP, and we have eBPF programs that will reject or allow the TCP connection based on whether you're allowed to do it.”
Bubna: Auto-research is hyperparameter tuning guided by model intuition, not architecture changes
“I, so the stuff I've seen people do with it is nowhere on the architecture level. It's pretty much tweaking parameters, but it's basically a high proprietor sweep that's guided by some sort of model intuition. So it's like much more efficient than whatever oth…”
Bubna: AI agents struggle to reason through logs and observability
“I think the things that sometimes agents struggle with without right guidance and a skill is how to use the rest of our observability. Like, how to, something is failing, like, How do you look at the logs and then update the right thing? It's sort of reasoning…”
Bubna: Batch compute demand comes mainly from non-LLM workloads like computational biology
“The demand that we see for something like that is actually not for LLMs. Although sometimes people want to run evals and do synthetic data prep and there it makes sense. But it's from a lot of non LLM companies like people who are doing computational bio, like…”
Bubna: Thousands more companies will post-train and deploy open-source models
“I think for example, LM inference, thousands more companies are going to post train their own models and deploy open source models for inference.”
Bubna: Production AI agents require specialized sandboxes for compute and networking control
“You need a lot more control over your compute primitive on things like what sort of, how do you persist different files that the agent has access to, and how do you snapshot and restore? How do you control the networking? Maybe you want GPUs. When you get to t…”
Modal CTO: Suno runs 100% of inference on Modal
“They use modal for all their inference, and that's because they have like a custom, they have completely custom model architecture, and that means that they have to be at the code level and tweak things that are not Yeah, it's an API.”
Bubna: Modal is bullish on CI as coding agents run more tests
“We're very bullish and modal on the CI market as well because it has, There's more agents coding agents they're gonna run a lot more CI, and the preventives there can be much better.”
Bubna: AI Agent Developers Prefer TypeScript Over Python
“And the interesting thing with, like, the agent stuff is people use their TypeScript SDK a lot more because they're not actually doing anything that needs ML.”
Bubna: AI agent hallucinations are actionable product feedback
“Sometimes it makes sense, like, if they're reaching for this thing, it's product feedback, like, give it to them.”
Bubna: Modal Refocused Its SDK Team on Agent Experience
“We've actually changed our SDK team to think about agent experience, sort of developer experience”
Bubna: Modal embedded an engineer at Cognition to reduce communication latency
“We sent him over because the latency of communication was too high otherwise.”
Bubna: Modal built agent sandboxes and published smol-developer recursive loops early
“We built sandboxes in May of the year before anyone knew this was going to be a thing. And the first example we published was we took a small developer and put in a loop so the agent can iterate on itself.”
Modal CTO: Production scale requires elastically scaling 1,000 to 1,500 GPUs quickly
“There it's not about scaling from zero to one, but it's how do we scale really elastically from, like, thousand to 1500 GPUs very quickly in, in a given region.”
Bubna: Modal Uses GPU Snapshotting to Speed Up Cold Starts
“We've incorporated GPU snapshotting to the product so we can actually take the GPU state, like your Torch compiler model, snapshot it, and the next call starts way faster.”
Bubna: Speculative decoding has zero impact on model output quality
“So there's no drop in quality performance, because you're always, you're never accepting a token that's a big model.”
Bubna: Ramp Inspect Succeeded Using Modal Snapshotting and Fast Scaling
“Ramp Inspect was a great example of a background agent that was really successful because they were able to use some of the primitives like snapshotting and fast scaling to just have something that feels really reactive and works well.”
Bubna: Modal's capacity pool spans 17 cloud providers
“We've built this capacity pool that spans 17 cloud providers. So we're very good at running on various kinds of cloud capacity across the world.”
Modal sandboxes support multi-container sidecar pods
“So actually, if you want Docker Compose our sandboxes now support this thing called Sidecarves. So you can, a sandbox is actually a pod of containers, and you can run multiple containers in a sandbox.”
Modal built an IPv6 overlay network for private container addressing
“We have this thing called I-SXPN, which we haven't talked about which is this, like, overlay network using IPv six addresses so if modal containers within the same workspace when this is enabled, can actually address each other using this private IPv six addre…”
Bubna: Transferring RL weights is fundamentally an OS memory problem
“Like the way you move around your KV cache and how efficiently you can do it, how efficiently you move your weights from your training GPUs to your inference GPUs in RL is, there's a lot of degrees of freedom, and it is basically a systems problem of Moving me…”
Bubna: Modal provides approximately 3 Tbps internal networking bandwidth
“And we have I think like three terabit per second internal networking, which is the standard that's needed.”
Bubna: Modal multi-node training targets post-training, not large-scale pre-training
“And we're not going for obviously like large scale pre-training runs. The thing that we've built multi-handle training for is we see a lot of smaller scale post-training like people are post-training like medium-sized fun models so they can get higher quality …”
Bubna: Modal uses autonomous agents to run internal inference optimization sweeps
“Internal both training and inference teams actually use this sort of the general shape of this quite a bit. Like we have this one internal repo called auto inference, which essentially we've automated our own FDE efforts using this harness, which is the agent …”
Bubna: Modal is building a cheaper 24-hour batch compute tier
“One of the things we're building now is, like, a way for customers to get if they don't care about latency like, get much cheaper pricing and they'll get results back in, like, next 24 hours or something. Like a batch tier, essentially.”