KV cache
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Dec 23, 2024 by Dylan Patel · across every show →
Everything said about KV cache, oldest first
Dec 23, 2024 neutral
Patel: Reasoning models like OpenAI o1 increase compute costs by 50x
“When I do this with O-one, right, because it's doing that thinking phase of 10,000... It spends a lot of memory on generating this KV cache and reading this KV cache constantly. Now the maximum batch size, i.e. Concurrent users I can have, is a fraction of tha…”