GPU HBM
topic on 2 shows · 2 statements across 2 episodes
the Y Combinator Startup Podcast 
Latent Space
2 statements about GPU HBM, every show
Stuart: Multi-GPU optimization requires overlapping compute with remote memory prefetching
“For multi-GPU kernels a simple, a similar idea applies, except that you're overlapping computation with communication with other GPUs, such that when the current computation is done, the data for next computation is ready and fetched from remote GPU HPMs.”
Biderman: Processing a Wikipedia article in Llama 70B consumes 80GB HBM
“If you take a Lama, a 70 B model, and you load one article from Wikipedia, which is a few tens of kilobytes, and you have the model read this The brain state of the model when reading this few tens of kilobytes is like, 80 gigabytes. 80 gigabytes on, on the HB…”