HBM bandwidth
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Sep 30, 2025 by Nick Joseph · across every show →
Everything said about HBM bandwidth, oldest first
Sep 30, 2025
Joseph: Inference needs more HBM bandwidth while pre-training is FLOPs-intensive
“Inference as a workload in general. Tends to require more HBM bandwidth. You end up doing you sort of the simplest form of sampling since you're going one at a time. You have to load all the weights for every token. And that means you might want a lot of HBM b…”