GPUs cannot handle unstructured sparsity as efficiently as Cerebras hardware
Sarah Chieng · [Paper Club] Weight Streaming on Wafer-Scale Clusters (w/ Sarah Chieng of Cerebras) · Dec 7, 2024 · at 38:26
Sarah Chieng of Cerebras compares how GPUs and Cerebras wafer-scale architectures handle fine-grained unstructured sparsity versus structured sparsity.
0:00 / 0:14exact quote · 14.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So both cerebris and GPUs can handle structured sparsity But GPUs are not designed to handle unstructured sparsity, whereas what I've just mentioned before is able to handle this unstructured sparsity.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Sarah Chieng
Cerebras WSE-3 runs Llama inference 70x faster than NVIDIA GPUs
“Cerebris came out that the wafer scale engine three can serve llama 70 B at 2.1 thousand sorry, 202,100 tokens per second and serves llama four or five B at nearly 1000 tokens per second. So this, you know, to give you an understanding, like this is about 70 t…”
Cerebras avoids model parallelism in production due to communication overhead
“There's a lot of communication overhead with model parallelism. You have to share activation tensors, and that is why in this paper and, you know, in production, Cerebra's focus on data parallelism. So all of this is mentioned in the paper as well, but model p…”
No competing AI framework disaggregates model storage from compute like Cerebras
“Like basically not, no one is doing anything close to where you're disaggregating. Model storage from compute. And none of these examples above do that either.”
Cerebras WSE-3 per-core SRAM eliminates central memory bandwidth bottlenecks
“So what Cerebrus has done for the wafer scale engine three is that instead of storing all these weights and values, weights and values off chip, Cerebrus stores everything on chip in SRAM. So every single one of the cores on the wafer scale engine three has it…”
Cerebras WSE-3 features 900,000 cores, 44GB SRAM, and 4 trillion transistors
“And so the wafer scale engine three, as I mentioned, 900,000 cores, 44 gigabytes of SRAM, four trillion transistors, and I do add a note here that the paper focuses on wafer scale engine two, and so the wafer scale engine three is, you know, just an upgraded v…”