inference performance
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Oct 1, 2025 by Andrew Feldman · across every show →
Everything said about inference performance, oldest first
Oct 1, 2025
Feldman: Memory bandwidth is the primary bottleneck in AI inference performance
“Inference performance comes from memory bandwidth and the memory bandwidth is the limiting factor. In inference performance. Remember, in order to generate a token, to generate a word, all the weights have to move from memory to compute. If you're constrained …”