kvcash
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Sep 19, 2024 by Umar Jamil · across every show →
Everything said about kvcash, oldest first
Sep 19, 2024 neutral
Jamil: KV Cache Prefilling Scales Quadratically While Token Generation Is Linear
“Token generation, which means generating one token using whatever is in the kvcash, is linear with respect to whatever is in the side of the kvcash. Prefilling the kvcash is quadratic, and mostly because it's quadratic, it's very expensive, so we are talking a…”