Memory Bandwidth
topic on 4 shows · 9 statements across 7 episodes
the Y Combinator Startup Podcast 
Latent Space 
Invest Like the Best 
20VC
9 statements about Memory Bandwidth, every show
Smulyanski: On-die SRAM accelerators excel at LLM decode due to high bandwidth
“The SRA machine basically keeps the entire weight matrix in SRA memory on DAI, so the, you got a lot more bandwidth, right, because it's on chip, right, so you can access, you know bytes over cycles, right, the chip interconnect is also fast, you can go, like …”
Smulyanski: GPU throughput drops sharply at low concurrency due to kernel overheads
“The moment that you start basically going to lower concurrency because you want better interactivity and better latency, Right? The performance the throughput drops. And it drops very sharply because all of a sudden you have a lot of, like, smaller kernels, yo…”
Baker: LLM pre-fill is capacity-bound while decode is bandwidth-constrained
“And that is fundamentally a memory capacity bound problem. Decode is the process of generating new tokens, and that is memory bandwidth constraint.”
Feldman: Memory bandwidth, not compute speed, limits GPU AI inference
“And that includes memory, which has, for inference, is the fundamental limiter for the GPU architecture. And so it doesn't matter how much faster the chip goes. It matters how much faster the memory bandwidth is.”
Feldman: Memory bandwidth is the primary bottleneck in AI inference performance
“Inference performance comes from memory bandwidth and the memory bandwidth is the limiting factor. In inference performance. Remember, in order to generate a token, to generate a word, all the weights have to move from memory to compute. If you're constrained …”
Sohmers: AI hardware over-indexes on raw FLOPS instead of memory bandwidth
“Everyone else was focusing on the wrong things. They were just trying to have more and more flops when memory bandwidth, memory capacity were the real, real bottlenecks.”
Sohmers: Positron AI hardware achieves 93% of theoretical memory bandwidth
“And so our fundamental architecture is enabling us, you know, today with hardware that we're shipping right now to be achieving, you know, 93% of the theoretical memory bandwidth of our device consistently across all use cases.”
Patel: Running LLaMA-70B at reading speed requires 2.1 TB/s memory bandwidth
“Hey, to run Llama's seventy billion requires two terabytes a second of memory bandwidth, 2.1, at reading, human reading speed.”