Unoptimized GLM-5.2 delivers a baseline 30 to 40 tokens per second
Philip Kiely · Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten · Aug 3, 2026 · at 37:05
Philip Kiely (Baseten) discusses baseline versus optimized inference throughput on large language models.
0:00 / 0:29exact quote · 29.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So let's say you have, as a reasonable baseline, 30 or 40 tokens per second. You can achieve 10 X that. So like on GLM 5.2 if you want to get unquantized perhaps on hoppers even and you're just using an off the shelf inference engine with no particular optimizations, no, no speculator nothing, nothing extra around like KV routing, no disaggregation. You, you're probably, yeah, looking at that, like, 30 to 40.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Philip Kiely
Quantization, speculative decoding, and prefill-decode disaggregation each yield 2x speedups
“Going from BF-sixteen to NVFP four is, it's not quite a two X, right? It's like, I think it's about like 30, 30 to 40% from 16 to eight, and then another 30 to 40% multiplied from eight to four. So that doesn't quite get you a two X, but like roughly a two X. …”
Leading AI agents will utilize continuous inference-to-training learning loops within two years
“And I think within a few months to a couple of years, like a lot of, Leading agent builders are going to have these loops, like, really up and running in production, where you are doing inference, learning from the inference.”
AI customers migrate from pay-per-token APIs to dedicated GPUs upon finding fit
“I think that we've increasingly seen a lot of demand for the sort of pay per token APIs just because everyone wants to try open models. And then once they find a use case that's really sticky then they move over to Dedicated.”
Open-source inference engines often receive model weights before official launches
“Getting to the point of I can make a token out of this model is not that hard because generally the open source inference engines, your VLMs, SGLangs of the world oftentimes even receive weights ahead of time maintainers do, or the people making the model merg…”