transformer inference

“Token generation, which means generating one token using whatever is in the kvcash, is linear with respect to whatever is in the side of the kvcash. Prefilling the kvcash is quadratic, and mostly because it's quadratic, it's very expensive, so we are talking a…”

“So the second level of this is that matrix vector multiplication is basically uncacheable. When you're doing transformer inference, matrix A is the weights of your model. And so if you're talking about model weights that are tens of gigabytes, hundreds of giga…”

“Everyone else was focusing on the wrong things. They were just trying to have more and more flops when memory bandwidth, memory capacity were the real, real bottlenecks.”