Transformer Inference
topic on 1 show · 3 statements across 2 episodes
3 statements about Transformer Inference, every show
Sohmers: Matrix-vector multiplication in transformer inference is fundamentally uncacheable
“So the second level of this is that matrix vector multiplication is basically uncacheable. When you're doing transformer inference, matrix A is the weights of your model. And so if you're talking about model weights that are tens of gigabytes, hundreds of giga…”
Sohmers: AI hardware over-indexes on raw FLOPS instead of memory bandwidth
“Everyone else was focusing on the wrong things. They were just trying to have more and more flops when memory bandwidth, memory capacity were the real, real bottlenecks.”
Jamil: KV Cache Prefilling Scales Quadratically While Token Generation Is Linear
“Token generation, which means generating one token using whatever is in the kvcash, is linear with respect to whatever is in the side of the kvcash. Prefilling the kvcash is quadratic, and mostly because it's quadratic, it's very expensive, so we are talking a…”