HBM Bandwidth
topic on 1 show · 1 statements across 1 episodes
the Y Combinator Startup Podcast
1 statements about HBM Bandwidth, every show
Joseph: Inference needs more HBM bandwidth while pre-training is FLOPs-intensive
“Inference as a workload in general. Tends to require more HBM bandwidth. You end up doing you sort of the simplest form of sampling since you're going one at a time. You have to load all the weights for every token. And that means you might want a lot of HBM b…”