memory bandwidth

“Hey, to run Llama's seventy billion requires two terabytes a second of memory bandwidth, 2.1, at reading, human reading speed.”

“And so our fundamental architecture is enabling us, you know, today with hardware that we're shipping right now to be achieving, you know, 93% of the theoretical memory bandwidth of our device consistently across all use cases.”

“Everyone else was focusing on the wrong things. They were just trying to have more and more flops when memory bandwidth, memory capacity were the real, real bottlenecks.”

“Inference performance comes from memory bandwidth and the memory bandwidth is the limiting factor. In inference performance. Remember, in order to generate a token, to generate a word, all the weights have to move from memory to compute. If you're constrained …”