Aug 3, 2026 · 1h 42m · latent-space
Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Baseten engineers Philip Kiely and Ali Taha join hosts Shawn Wang and Alessio Fanelli to dissect the state of AI inference engineering, covering speculative decoding, NVFP4 quantization, distributed cluster architectures, multimodal scaling, and autonomous self-optimizing GPU kernels.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 7.6% of the talking time here. How this is scored →
The hosts as informed peer 5.0 Guest teaching 6.0 Guest disagreement 1.3 The hosts pushing back 1.6
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 58:14 Ali dismisses fused mega kernels
Ali aggressively rejects the popular industry trend of fused mega kernels, stating he is very bearish and noting that in production, modular kernels consistently outperform them.
Hardest push from the hosts ▶ 1:06:29 Swyx defends custom ASICs against GPU dominanceSwyx directly challenges Ali's claim that GPUs make dedicated ASICs irrelevant, pointing to custom mega-scale lab chips and wafer-scale hardware that optimize beyond GPU silicon constraints.
Biggest teaching moment ▶ 31:53 Ali explains mathematical quantization error cancellationAli walks through empirical research and mathematical proofs demonstrating how quantizing more complementary layers can actually increase overall model fidelity by canceling out distribution errors.
The host holds their own ▶ 34:44 Swyx connects inference gains to 1970s quantitative financeSwyx demonstrates deep domain knowledge by drawing a historical parallel to Andrew Lo's statistical arbitrage research, comparing modern inference speedup compounding to 1970s quantitative finance margin compression.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Studio Welcome and the Waterloo Handle | 3 | 5 | 0 | 1 | Alessio sets up the discussion by asking what occurs under the hood when a user submits a massive 200,000-token prompt. Philip explains the inference mechanics across cache-aware routing, disaggregated prefill/decode instances, and speculative decoding. | |
| Dedicated Deployments and Speculative Decoding Economics | 4 | 6 | 0 | 1 | Swyx highlights the economics of renting dedicated boxes versus paying per token, while Ali explains how speculative decoding draft models act like attached parasites fine-tuned on domain-specific traffic. | |
| Tool Calling Challenges and Structured JSON Output | 5 | 4 | 0 | 2 | Swyx and Philip examine structured outputs, BNF grammars, and JSON parsing constraints. Philip clarifies that LLMs merely produce formatted suggestions that runtime state machines enforce. | |
| Day-Zero Model Support and Productionization Work | 4 | 5 | 0 | 1 | Philip and Ali detail the extensive engineering required to make day-zero open-source model releases production-ready, including NVFP4 recalibration and speculator training. | |
| Retrofitting Multimodal Vision onto LLMs | 5 | 5 | 0 | 1 | Ali and Philip describe retrofitting Kimi's vision encoder onto GLM-5.2 via a trained projector layer. Swyx and Alessio draw parallels to foundational multimodal research like LLaVA and MetaCLIP. | |
| Model Surgery, Mode Collapses, and Cluster Race Conditions | 5 | 7 | 1 | 2 | Ali explains how subtle GPU kernel race conditions and cluster interconnect differences cause non-deterministic mode collapses. Swyx questions why computational graphs do not catch these synchronization bugs. | |
| Mathematical Quantization and Layer Error Cancellation | 4 | 8 | 1 | 1 | Swyx asks whether quantization is strictly lossy. Ali details internal research proving mathematically that quantizing specific complementary layers causes rounding errors to cancel each other out. | |
| Compounding Performance Gains Toward 10x Inference Speedups | 6 | 5 | 0 | 1 | Philip explains how stacking multiple 2x optimizations yields 10x throughput jumps. Swyx connects this to historical quantitative finance research by Andrew Lo on compressing market inefficiency margins. | |
| Engineering Trade-offs, Blackwell Deployments, and NVIDIA Dynamo | 6 | 5 | 1 | 2 | Alessio and Swyx push on the concrete engineering steps required to max out a B200 node. Philip clarifies that NVIDIA Dynamo functions as a distributed developer toolkit for KV routing rather than an auto-optimizer. | |
| The Evolution of Speculative Decoding Architectures | 5 | 6 | 1 | 2 | The conversation explores speculative decoding evolutions from Medusa to Eagle and Tri Dao's SpecSpecDecode. Philip highlights the resource competition when running recursive draft models on shared hardware. | |
| Data Center Scale vs. Edge and Local Device Inference | 5 | 6 | 1 | 1 | Philip and Ali contrast data center scale optimizations with edge inference constraints, illustrating why techniques like TurboQuant provide huge wins on MacBooks but harm B200 GPU performance. | |
| Distributed Inference: Tensor, Expert, and Pipeline Parallelism | 6 | 6 | 1 | 2 | Philip breaks down tensor, expert, and pipeline parallelism strategies across GPU clusters. Swyx probes why more models do not follow AI21 Jamba's approach of tailoring architecture dimensions to specific chips. | |
| A Bearish Perspective on Fused Mega Kernels | 4 | 7 | 7 | 1 | Ali delivers a contrarian take against the industry trend of fused mega kernels, arguing they are too difficult to maintain and that modern hardware developments like NVIDIA Rubin make them obsolete. | |
| Hardware Evolution: Anticipating NVIDIA Rubin and Systems Architecture | 4 | 6 | 1 | 1 | Philip reflects on multiple hardware launch cycles, predicting that NVIDIA Rubin will shift inference engineering from micro-level CUDA kernels to holistic cluster-level systems infrastructure. | |
| The Debate on GPUs Becoming Specialized ASICs | 7 | 5 | 6 | 7 | Ali contends that GPUs are evolving into domain-specific ASICs, challenging the premise of dedicated AI ASIC startups. Swyx vigorously pushes back, highlighting vertically integrated custom chips like OpenAI/Broadcom and wafer-scale architectures. | |
| Managing Extreme Model Scale and Multi-Trillion Parameter Deployments | 5 | 6 | 0 | 1 | Alessio and Philip walk through the hardware requirements for multi-trillion parameter deployments like Kimi 3T, calculating exact NVFP4 weight memory versus KV cache headroom on GB300 nodes. | |
| The Quadratic Attention Bottleneck in Video Diffusion | 4 | 8 | 2 | 1 | Ali breaks down the computational roadblocks of video diffusion models, showing how the quadratic attention bottleneck over tens of thousands of 3D latent tokens forces severe quality degradation. | |
| Autoregressive Video vs. Chunk Stitching and Drift | 5 | 7 | 3 | 1 | Ali and Philip compare autoregressive video generation against video chunk stitching, explaining the severe visual drift and darkening that occurs when extending open-source video models. | |
| Multimodal Paradigms: Speech Tokens and Text Diffusion | 7 | 5 | 2 | 3 | The panel explores speech tokens and text diffusion architectures like Diffusion Gemma and Mercury. Swyx advocates for rethinking diffusion APIs around context-free guidance and structural generation rather than conversational chat. | |
| The Deep Convergence of Training and Inference Workflows | 5 | 6 | 0 | 1 | Ali and Philip discuss the convergence of training and inference, highlighting how reinforcement learning rollouts and quantization-aware distillation require inference teams to operate robust training pipelines. | |
| Self-Optimizing Models and Autonomous Kernel Engineering | 5 | 7 | 1 | 1 | Ali reveals an internal experiment where GLM-5.2 analyzed SGLang performance profiling traces and wrote autonomous GPU kernels to eliminate its own runtime bottlenecks. | |
| Future Frontiers: Faster NICs and Inter-Node KV Transfers | 4 | 7 | 1 | 1 | Ali outlines his top bottleneck for the future of inference, explaining that slow NIC speeds constrain inter-node KV cache transfers compared to ultra-fast on-chip HBM bandwidth. | |
| Continual Learning: Weight Modification vs. KV Cache Compaction | 6 | 7 | 1 | 2 | Swyx brings up continual learning paradigms. Ali explains why editing MLP weights fails on multi-hop reasoning tasks, making infinite KV cache compaction the superior path for lifelong learning. |
Statements from this episode (51)
Hourly GPU instances are cheaper than pay-per-token for high-volume AI inference
“And more often than not, it's like way cheaper if you're pushing like millions of tokens per hour, if you just pay per hour instead of pay per token.”
AI customers migrate from pay-per-token APIs to dedicated GPUs upon finding fit
“I think that we've increasingly seen a lot of demand for the sort of pay per token APIs just because everyone wants to try open models. And then once they find a use case that's really sticky then they move over to Dedicated.”
Unclosed JSON syntax is the primary cause of model tool hallucinations
“The challenge with tool calling more and more seems to be that the companies want certain tool calling, which is a very sensitive thing to train. And because you're dealing with all of the JSON outputs, if it doesn't like close the end of the request in a very…”
Open-source inference engines often receive model weights before official launches
“Getting to the point of I can make a token out of this model is not that hard because generally the open source inference engines, your VLMs, SGLangs of the world oftentimes even receive weights ahead of time maintainers do, or the people making the model merg…”
Training speculative decoding models requires hidden states from the base model
“Now, with speculators today, you need to train the speculator using the base model itself, because you're getting hidden states out of the model from running inference on these specific prompts, and that is the training data you use to create the speculator.”
DeepSeek models are the hardest to support due to architectural novelties
“I think that, like, obviously the DeepSeq models tend to be the most challenging as they have, like, the most novel architectural stuff going on model over model.”
Modifying base LLM weights for vision degrades original text performance
“You don't want to mess with the model weights because you run a chance of making the model dumber at something else for the purpose of giving it vision.”
Baseten's vision-retrofitted GLM-5.2 scored 56% on MMLU Pro without text degradation
“It's not, you know, it got to a 56% on MMLU Pro, I think, so not, not quite Frontier, but if you're running this model, you haven't suffered any loss on your GLM-Five-II quality.”
Retrofitting Grouped Query Attention layers cures inference bottlenecks and maintains acceptance
“So we find it better to, like, okay, we're going to replace this, you know, we're going to replace this layer with a layer from another model that's using, like, GQA, for instance. And then just for the right training, you can get it to have the same acceptanc…”
Baseten terminates or reprocesses generations after four identical repeating tokens
“We have, like, in our endpoint, like, if a model was to output the same exact token, like, four plus times, we just call the generation, we say, like, oh, sorry, this, like, try again, or, like, we will re-process the request. Because we know then, like, if it…”
Most inference optimizations like KV caching are lossless, unlike quantization
“Most inference optimizations are lossless. KV caching, for example, you are just recomputing or preventing recomputing the same values. Speculation, of course, If a draft token is wrong, it gets rejected. The main lossy optimization is quantization”
Quantizing more layers can actually improve model fidelity via error cancellation
“It is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out. And so what Joshua showed in his mathematical proof where he had like a verifier in is that you can predict which …”
Inference optimization is only solved once researchers report mere 1% speedups
“Like, you'll, you'll know that influence is pretty much solved when researchers start publishing about how they got one percent faster at something.”
Unoptimized GLM-5.2 delivers a baseline 30 to 40 tokens per second
“So let's say you have, as a reasonable baseline, 30 or 40 tokens per second. You can achieve 10 X that. So like on GLM 5.2 if you want to get unquantized perhaps on hoppers even and you're just using an off the shelf inference engine with no particular optimiz…”
Software optimizations yield 2x to 4x inference speedups on identical hardware
“Yeah, then you're looking at, like, a two to four X improvement, depending on the inference optimizations.”
Quantization, speculative decoding, and prefill-decode disaggregation each yield 2x speedups
“Going from BF-sixteen to NVFP four is, it's not quite a two X, right? It's like, I think it's about like 30, 30 to 40% from 16 to eight, and then another 30 to 40% multiplied from eight to four. So that doesn't quite get you a two X, but like roughly a two X. …”
NVIDIA or model creators will always publish open quantized checkpoints
“There's always going to be like an open source quantized checkpoint. NVIDIA is going to push one out if no one else does. You usually the providers will have their own spec tech that they've trained as well. You don't need to train your own spec tech. You can …”
NVIDIA Dynamo is a developer toolkit, not an out-of-the-box performance optimizer
“I would think of Dynamo as less of a sort of out of box system and more of a toolkit for building with. So when we talk about doing KV aware routing, when we talk about doing KV out offloading, when we talk about doing PD disaggregation, Dynamo fundamentally i…”
Eagle remains widely used despite newer speculative decoding methods like DeFlash
“The speculation space has moved much faster than everything else. So, yeah, even at the time that I wrote the book Medusa, I very much included as a way for people to understand how the, how, how the space evolved rather than what the most modern technique is,…”
Speculative draft models are typically one-sixteenth the target model's size
“Like for Minimax, it's, yeah, yeah. It's like one layer. It's like one 16th of the original model usually.”
Speculative decoding creates hardware resource contention and engine orchestration complexity
“The thing with speculators is one of the practical constraints on using them is that you do have to run a small model on the same hardware that you're running the big model on. There is a orchestration and resource competition problem inherent in that. And tha…”
Local AI optimizes for memory fit; data centers optimize for speed
“The difference between inference engineering for the data center and for production workloads versus inference engineering for local AI is that we start with fundamentally like different constraints and different goals. With local AI, it's how do I fit this mo…”
TurboQuant is inefficient on high-bandwidth data center GPUs like NVIDIA B200
“TurboQuant would not be like, it would not be used. Like Nvidia made it clear that this is not a good optimization. And we've seen it firsthand where the overhead of doing dequantization, quantization of You know, in the kernel itself, the turbo-quant kernel, …”
Tensor parallelism is a poor fit for local multi-device AI setups
“Tensor parallelism is not a good fit for local AI because it assumes a very high bandwidth interconnects like NVLink was, you know, They might be forced to do something like pipeline parallelism, which we're never going to do unless we're doing some kind of mu…”
All modern AI models requiring multi-GPU parallelization are Mixture-of-Experts
“Effectively, all models today are MOE models that are, you know, at least all models large enough that you would care to parallelize them across multiple GPUs.”
Pipeline parallelism is only necessary for multi-node AI model inference
“The only reason you would have to do pipeline parallelism which is where you separate, like, different layers, and you put, like, half the layers on one hardware and half on another, is if you are forced to do multi-node inference because a model is bigger tha…”
Optimal inference parallelism cannot be mathematically calculated; it must be auto-tuned
“And with training, it's more of like a math, like you can run the math and see the flops and maximize it. With inference, it's more of like an auto-tuning, like GPU kernel auto-tuning... You shadow the same traffic, like real traffic, and you just see which co…”
Most open AI research labs avoid hardware-specific model architecture optimization
“Most open labs are trying to make models that can be run on as wide of hardware as possible rather than targeting just like a single chip.”
Fused GPU kernels cannot eliminate inter-GPU communication in tensor parallelism
“Well, I mean, like a fused kernel can't save you. Like here with the tensor parallelism, you're the half, the matrix is one GPU and the other half is on another. And if I need the entire matrix in order to do like a non-linear operation on the next step, which…”
Companies developing fused mega kernels rarely run them in production
“Even the companies that have worked at or people that have spoken to who work at companies that do fused mega kernels, they, Very, very often don't end up running those in production because the TRTL and modular kernels that will launch are faster because you …”
NVIDIA's Vera Rubin GPU architecture effectively renders mega kernel research obsolete
“The GPU is, is It's designed in such a way that it basically kills megakernels. You don't need to use megakernels that much anymore. So it seems like that entire research field goes into, like, won't be continued, but yeah.”
NVIDIA Rubin will shift inference engineering toward traditional hardware infrastructure challenges
“I think that themes around like KV cache offloading, KV aware routing, and disaggregation are going to be substantially more important in the Rubin era, which means that inference engineering becomes not just a like CUDA kernel problem, but also like a very tr…”
AI ASIC startups are doomed as NVIDIA GPUs become domain-specialized
“How can you look at this trend and then still be bullish on companies that are coming up with Asics for AI.”
NVIDIA's Rubin is the first GPU architecture designed entirely for modern LLMs
“Ruben's honestly the first chip that was fully built in that world. And so you can see a lot of the understanding of the shape of the workload that this chip's going to be asked to do in the way it's designed.”
Custom $50B ASICs become economically rational for $500B AI training runs
“If you have a trillion dollar or five hundred billion dollar training run, then take fifty billion of that and make an ASIC. Like, it's fine. Like you will get more than 10% efficiency from the ASIC. And like, that makes sense.”
Fitting a 2.8-trillion parameter model on one node requires eight GB300s
“You need GB 300 to fit it on a single node. It's simple math. NVFP four, 2.8 trillion parameters 1.4 terabytes. The GB 300 have 288 gigabytes each. So across eight of those you have enough room For the model”
Current inference hardware capacity sets the parameter ceiling for new foundation models
“Like, people understand the ceiling that you have in terms of how many parameters of a model you can run, given the sort of latest inference hardware, and that kind of forms a ceiling.”
Open-source LLMs have reached closed parity, but video models lag far behind
“The difference between the best open source LLM and the best open closed source LLM is very small. Like it used to be six months. I don't think it's six months anymore. I think it's like basically almost unparative. Video models are definitely not, there's a h…”
Long-form AI video generation must switch to autoregressive architectures over diffusion
“Autoregressive video seems to me like that is the bet that the future is going to be making. But there are no good open source autoregressive video models out there today. And that seems to be, if you want to get like an hour movie, if you want to see video mo…”
Every current autoregressive video model has terrible output quality compared to diffusion
“The downsides is every single
Autoregressive video model is shit.
It's just terrible quality.
If you, I like, it's just, if you put the quality of any opens like one two or two versus any other autoregressive model, you can see like a video generated by one tw…”
Future long-form video generation will likely hybridize autoregressive and diffusion techniques
“So if we think about the architecture that's going to get us there to these longer, richer sequences, it's probably, like you said, going to be a mix of the auto aggressive and the diffusion working together to do what each piece is good at.”
Modern speech models operate autoregressively by adding waveform tokens to LLM vocabularies
“Speech is autoregressive. You effectively I mean, this was even back with, like, the Orpheus architecture a year and a half ago. You just add a bunch of waveforms to the vocabulary so that the LLM can output tokens that represent those waveforms, and then you …”
Midjourney's David Holtz explored text diffusion to storyboard entire movies
“David Holtz from Midjourney was investing in text diffusion. I don't think anything came out of it, but like the idea was that you can storyboard a long movie and then you can generate the scenes with video, normal video gen.”
Leading AI agents will utilize continuous inference-to-training learning loops within two years
“And I think within a few months to a couple of years, like a lot of, Leading agent builders are going to have these loops, like, really up and running in production, where you are doing inference, learning from the inference.”
Dynamic system adjustments consistently outperform static configurations in AI inference setups
“Any kind of dynamic adjustment is going to beat a static configuration across, you know, your exact config, across your speculator, across that kind of thing.”
Writing deployment configs for arbitrary Hugging Face models is now one-shot-able
“One big part of my job a couple years ago was for any arbitrary model that came out on Hugging Face, writing a config for it and kind of getting it up and running, and now the get it up and running config is, is one-shot-able and so, you know, I don't have to …”
GLM-5.2 autonomously wrote and guided production GPU kernels for Baseten's inference engine
“Some of the GPU kernels that were on GLM-Five-two within our inference engine is written by GLM-Five-two. And the trace and the kernels were guided by GLM-Five-two as the driver.”
Network interface card speed is the primary bottleneck in large-scale AI serving
“I think the answer is just faster next, like faster network chip communications. It seems to me that like more and more memory is the bottom, like you want to have larger models. Right now, when you're doing serving at large, you have to transfer KV cache from…”
Ultra-fast network interface cards could deliver 100x speedups in disaggregated AI inference
“If you were to somehow be able to, in like this theoretical dreamland, have extremely fast NICs, you could, in theory, spare that HBM and you could just transfer KVCache trans, like directly from one node to another. This would give you like almost a hundred X…”
Editing isolated facts in MLP weights fails to update multi-hop downstream reasoning
“So if you know that the best university in the world is Waterloo, then the answer should be Waterloo. But if I wasn't just one-shotting the question and I was to ask it to like, use its knowledge to think and then give me a second answer, or like, should I hir…”
KV cache compaction is the viable path to solve LLM continual learning
“If you're able to sort of make your KV almost infinite, and you're able to compact in such a way that you don't lose any of the knowledge. In that case, you can actually do a continual learning as you can actually solve continual learning. And this, as a, it's…”