Jan 19, 2025 · 57m · latent-space
DeepSeek V3, SGLang, and the state of Open Model Inference in 2025 (Quantization, MoEs, Pricing)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space podcast, Baseten leaders Amir and Yining Zhang explore the architectural breakthroughs of DeepSeek V3 and the technical evolution of the SGLang inference engine. They provide an in-depth breakdown of what it takes to operate mission-critical, dedicated inference infrastructure across multi-cloud enterprise environments in 2025.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 4.5 Guest teaching 5.6 Guest disagreement 0.9 The hosts pushing back 0.9
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 31:20 Calling out the naive production vLLM assumption
Amir forcefully rejects the prevalent industry misconception that simply wrapping an open-source serving engine like vLLM behind an API constitutes an enterprise-ready production inference deployment.
Hardest push from the hosts ▶ 49:16 Swyx disputes the demise of fine-tuningSwyx directly interrupts and pushes back against Amir's hypothesis that reasoning models will make fine-tuning obsolete, insisting prompt iteration is the actual operational substitute.
Biggest teaching moment ▶ 28:12 DeepSeek architectural breakthroughs in SGLangYining Zhang educates the hosts on how SGLang uniquely implemented Multi-Head Latent Attention and custom blockwise FP8 CUDA kernels before other major open-source frameworks adopted them.
The host holds their own ▶ 39:40 Alessio probes finite state machine constrained decodingAlessio demonstrates deep technical fluency in decoding mechanics by drilling into FSM transitions, jump-forward token compression, and potential accuracy trade-offs in structured outputs.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Technical and Hardware Challenges of DeepSeek V3 | 4 | 5 | 1 | 2 | Swyx probes into the hardware constraints of serving the 671B DeepSeek V3 model. Yining Zhang explains memory constraints and FP8 kernel implementations, while Swyx playfully pushes back on whether model debugging is complicated or merely slow. | |
| Model Sizing, Claude Migration Trends, and Hardware Sizing | 5 | 6 | 1 | 1 | Alessio asks why Llama 405B saw low adoption compared to 70B and where DeepSeek interest originates. Amir explains that demand is primarily driven by Claude migration due to rate limits, pricing, and latency needs rather than upgrades from smaller open-source models. | |
| Native FP8 Quantization and Dedicated Model Serving | 5 | 6 | 2 | 1 | Swyx brings up native FP8 quantization trends and compares Baseten's approach to competitors like Together and Fireworks. Amir clarifies Baseten's core architectural philosophy of providing dedicated rather than shared inference endpoints to preserve model fidelity and meet strict SLAs. | |
| MoE Architectures, Training Stability, and Industrial Trends | 5 | 6 | 1 | 2 | Swyx questions why major Western labs like Meta and xAI opted for dense models over MoE architectures throughout 2024. Yining Zhang details training instability issues where early MoE benchmark scores lagged behind dense counterparts during training runs. | |
| Baseten Consumption Pricing, BYOC, and Multi-Cloud Scaling | 3 | 5 | 0 | 0 | Swyx asks about navigating aggressive price competition in the open model space. Amir outlines Baseten's pure consumption pricing model and its enterprise multi-cloud orchestration where workloads horizontally scale across multiple cloud providers. | |
| Truss Packaging Library and Inference Engine Integrations | 4 | 5 | 0 | 0 | Alessio asks about the software packaging and engine stack powering DeepSeek V3 serving. Amir details their open-source packaging framework Truss and how they modularly integrate runtime engines like TensorRT-LLM and SGLang. | |
| Low-Latency Multi-Model Pipelines with Truss Chains | 4 | 6 | 1 | 1 | Alessio asks about the architectural evolution of Truss from simple microservices to complex workloads. Amir explains how the demand for low-latency composite AI workflows led to Truss Chains to eliminate inter-model network serialization overhead. | |
| Transparency, Reliability, and Enterprise Inference SLAs | 4 | 5 | 0 | 0 | Alessio asks whether enterprise clients care about the underlying serving engine. Amir emphasizes that enterprise and foundation model customers demand total transparency, rigorous tail latency guarantees, and strict geo-compliance over black-box endpoints. | |
| The Three Pillars of Mission-Critical Inference Workloads | 3 | 7 | 2 | 0 | Amir articulates the three pillars required for production AI inference, firmly rejecting the naive industry assumption that serving frameworks like vLLM alone constitute production-grade readiness. | |
| SGLang Origins, Radix Cache, and Performance Milestones | 4 | 6 | 1 | 0 | Alessio references the SGLang research paper and asks about its origin story. Yining Zhang recounts its evolution from a front-end language to a high-throughput serving engine designed to combine TensorRT-LLM performance with vLLM developer extensibility. | |
| Radix Attention, Prefix Caching, and Cache-Aware Load Balancing | 5 | 5 | 0 | 0 | Alessio asks about RadixAttention and fine-grained KV cache reutilization. Yining Zhang explains the performance advantages of block size one prefix caching, while Amir describes how Baseten built global cache-aware routing across multi-replica deployments. | |
| Constrained Decoding, FSMs, and XGrammar Integration | 6 | 6 | 1 | 1 | Alessio inquires about constrained decoding using finite state machines and token jumping. Yining Zhang and Swyx discuss the trade-offs and highlight the adoption of XGrammar over Outlines across modern inference engines. | |
| Speculative Decoding Realities, xAI Adoption, and SGLang Roadmap | 5 | 5 | 1 | 1 | Swyx notes xAI's adoption of SGLang for Grok and discusses speculative decoding algorithms. Amir points out that framework support is trivial compared to the complex downstream challenge of effectively training and aligning draft models. | |
| Reinforcement Learning, Reasoning Models, and the Future of Fine-Tuning | 6 | 5 | 2 | 4 | Amir suggests that long-term advances in reasoning models might eventually obsolete specialized fine-tuning. Swyx strongly challenges this assertion, arguing that prompt engineering is far more likely to replace fine-tuning than reasoning models alone. | |
| Enterprise Scale, Real-World Adoption, and Cursor Integration | 5 | 6 | 1 | 1 | Yining Zhang highlights Cursor adopting SGLang for DeepSeek V3, prompting Amir to reiterate the comprehensive infrastructure requirements for mission-critical enterprise workloads. Swyx advises Baseten to publish an EnterpriseReady-style architectural manifesto. |
Statements from this episode (18)
Zhang: DeepSeek-V3 is currently the leading open-source LLM
“Yeah, because DeepSeq VIII, I think, is currently considered the leading open source LLMs based on the benchmark results and the chat area results.”
Zhang: DeepSeek-V3 cannot run on a single 8xH100 GPU node
“You need, I think 671 gigabytes for the weights, and you also need an extra memory for the KV cache, so it's not possible to run that on H-one hundred.”
Zhang: Llama 405B sees very few enterprise users compared to 70B
“I think at the base time, something like LAMA-Seventy-B is more common. I think LAMA-Seventy-B has released the 400 zero five billion weights, but I think there are just a few users use that.”
Yining Zhang: DeepSeek V3 scores 94.6 on GSM8K, outperforming Llama 405B
“Yeah, I think even they use the FP-A to quantization, the benchmark result is very good, such as something like GSM-HK. The score is nearly 94.6. It's so high, you know. I think it's higher than every other open source AIM, even the LAMA 400 zero five billion.”
Zhang: Baidu and ByteDance Internal Models Use DeepSeek-Like MoE Architectures
“As far as I know, some companies such as Baidu or Baidu Dance, they are internal, the dominant AOM, they use the MOE architecture, and their ways, I think, is similar to the DeepSeq MOE model.”
Zhang: MoE Inference Optimization Will Be Essential in 2025
“So I think at this new year, the MOE inference optimization will be very essential, and yeah, Yeah, so important.”
Zhang: Meta Failed at Training MoE Models for Llama Series
“The reason why Lama open-sourced the MOE model, because I think they tried to train our MOE model, but they failed. So that, that's why they didn't open source MOE mode for Lama series.”
Zhang: SGLang outperforms vLLM and has better usability than TensorRT-LLM
“I think for the common use case, maybe not, not the DeepSeq VIII, for the common use case, I think SGLAN's performance is better than FLM, and its usability is better than TensorFlow TLM.”
Zhang: vLLM does not support DeepSeek MLA while SGLang does
“Something like DeepSeq V-II, they proposed attention parent named MLA, multi-latent attention, and I think SGLAN is the only framework to support that. Maybe LightLM and TRTM also support, but VLM doesn't support.”
Zhang: DeepSeek team officially recommends SGLang as its inference engine
“That's why SGLAN is the recommended LLM engine by the DeepSeq team.”
SGLang was the first framework to support prefix caching
“At 2024, January, they support something like Redix cache. It's a prefix caching technology. I think SGLAN is the first framework that supports prefix cache.”
Zhang: SGLang achieved 3x throughput over vLLM in mid-2024 benchmarks
“At that time, I think its performance is maybe three times, is throughout, put it, three times than FLM.”
Zhang: TensorRT-LLM is blazing fast but difficult to extend
“And TensorFlow RTM, I think it's blazing fast. Its performance is so good, but it's not easy to do some secondary development. If you want to add some new feature, it's a little hard.”
Zhang: SGLang achieves higher cache hit rates using block size of one
“Redix cache, I think it's the technology of the prefix caching, and it is a special case for something like block size is one, you know, for VLM or for other frameworks, they use something block size, 32, and SGLAN use the block size one. I think if you use th…”
Zhang: XGrammar outperforms Outlines and is integrated into TensorRT-LLM
“And I think Xgrammar's performance is better than the outline's, and also in the TensorFlow RTLM, the latest release, TensorFlow RTLM also integrates Xgrammar as the backend for the constructed coding.”
Zhang: SGLang creators Lianmin Zheng and Ying Sheng work at xAI
“Lian Ming and Yin are the XAI's member of the technical staff.”