Kvcash
topic on 1 show · 1 statements across 1 episodes
1 statements about Kvcash, every show
Jamil: KV Cache Prefilling Scales Quadratically While Token Generation Is Linear
“Token generation, which means generating one token using whatever is in the kvcash, is linear with respect to whatever is in the side of the kvcash. Prefilling the kvcash is quadratic, and mostly because it's quadratic, it's very expensive, so we are talking a…”