Inference Performance
topic on 1 show · 1 statements across 1 episodes
1 statements about Inference Performance, every show
Feldman: Memory bandwidth is the primary bottleneck in AI inference performance
“Inference performance comes from memory bandwidth and the memory bandwidth is the limiting factor. In inference performance. Remember, in order to generate a token, to generate a word, all the weights have to move from memory to compute. If you're constrained …”