GPU Kernels
topic on 2 shows · 3 statements across 3 episodes
3 statements about GPU Kernels, every show
Bubna: Speculative decoding accept length delivers multiplicative speedups over kernel tuning
“People talk a lot about, we made these kernels faster and whatnot, but improving kernel only give you like a few percentage points of improvement and increasing except length literally is a multiplicative decrease.”
Writing custom GPU kernels is a last resort in model optimization
“Well, kernel is a last resort. So first I go, oh, another kernel. And I try and find some way to go around it.”