Speculative Decoding
topic on 2 shows · 10 statements across 4 episodes
10 statements about Speculative Decoding, every show
Most inference optimizations like KV caching are lossless, unlike quantization
“Most inference optimizations are lossless. KV caching, for example, you are just recomputing or preventing recomputing the same values. Speculation, of course, If a draft token is wrong, it gets rejected. The main lossy optimization is quantization”
Quantization, speculative decoding, and prefill-decode disaggregation each yield 2x speedups
“Going from BF-sixteen to NVFP four is, it's not quite a two X, right? It's like, I think it's about like 30, 30 to 40% from 16 to eight, and then another 30 to 40% multiplied from eight to four. So that doesn't quite get you a two X, but like roughly a two X. …”
NVIDIA or model creators will always publish open quantized checkpoints
“There's always going to be like an open source quantized checkpoint. NVIDIA is going to push one out if no one else does. You usually the providers will have their own spec tech that they've trained as well. You don't need to train your own spec tech. You can …”
Speculative draft models are typically one-sixteenth the target model's size
“Like for Minimax, it's, yeah, yeah. It's like one layer. It's like one 16th of the original model usually.”
Speculative decoding creates hardware resource contention and engine orchestration complexity
“The thing with speculators is one of the practical constraints on using them is that you do have to run a small model on the same hardware that you're running the big model on. There is a orchestration and resource competition problem inherent in that. And tha…”
Bubna: Speculative decoding accept length delivers multiplicative speedups over kernel tuning
“People talk a lot about, we made these kernels faster and whatnot, but improving kernel only give you like a few percentage points of improvement and increasing except length literally is a multiplicative decrease.”
Bubna: Speculative decoding has zero impact on model output quality
“So there's no drop in quality performance, because you're always, you're never accepting a token that's a big model.”
Dan Fu: Proper speculative decoding yields 2x to 3x model speedups
“So if you do the speculative decoding right, you can get, again, two X, three X speed ups over, over, you know, just running a vanilla model.”
Agarwal: Distillation accelerates speculative decoding for large models
“Now, the thing is, the effectiveness of this method depends on how close the sampler, the small model is to the bigger model that we want to speed up, and actually distillation exactly fixes that, which is, by distillation, you can make things closer to each o…”
Agarwal: Google AI Overviews Uses Speculative Decoding With Distilled Models
“And this was actually used, so I guess the cool application of this I can mention is the next slide, which is AI overviews at Google. I don't know if people have seen this or this kind of thing. Like, there is this thing that comes up, and that actually uses t…”