on-chip cache
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Aug 18, 2025 by Thomas Sohmers · across every show →
Everything said about on-chip cache, oldest first
Aug 18, 2025
Sohmers: Matrix-vector multiplication in transformer inference is fundamentally uncacheable
“So the second level of this is that matrix vector multiplication is basically uncacheable. When you're doing transformer inference, matrix A is the weights of your model. And so if you're talking about model weights that are tens of gigabytes, hundreds of giga…”