Cerebras WSE-3 per-core SRAM eliminates central memory bandwidth bottlenecks
Sarah Chieng · [Paper Club] Weight Streaming on Wafer-Scale Clusters (w/ Sarah Chieng of Cerebras) · Dec 7, 2024 · at 9:51
Sarah Chieng of Cerebras explains why the Wafer-Scale Engine architecture achieves higher inference speed compared to GPUs like the NVIDIA H100.
0:00 / 0:40exact quote · 40.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So what Cerebrus has done for the wafer scale engine three is that instead of storing all these weights and values, weights and values off chip, Cerebrus stores everything on chip in SRAM. So every single one of the cores on the wafer scale engine three has its own SRAM. So each core has direct access To each of the values that it needs to do its computations. And this design eliminates the need to repeatedly fetch weights from a central memory, which then significantly reduces the memory bandwidth requirements and power consumption of the wafer scale engine.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Sarah Chieng
Cerebras WSE-3 runs Llama inference 70x faster than NVIDIA GPUs
“Cerebris came out that the wafer scale engine three can serve llama 70 B at 2.1 thousand sorry, 202,100 tokens per second and serves llama four or five B at nearly 1000 tokens per second. So this, you know, to give you an understanding, like this is about 70 t…”
Cerebras avoids model parallelism in production due to communication overhead
“There's a lot of communication overhead with model parallelism. You have to share activation tensors, and that is why in this paper and, you know, in production, Cerebra's focus on data parallelism. So all of this is mentioned in the paper as well, but model p…”
GPUs cannot handle unstructured sparsity as efficiently as Cerebras hardware
“So both cerebris and GPUs can handle structured sparsity But GPUs are not designed to handle unstructured sparsity, whereas what I've just mentioned before is able to handle this unstructured sparsity.”
No competing AI framework disaggregates model storage from compute like Cerebras
“Like basically not, no one is doing anything close to where you're disaggregating. Model storage from compute. And none of these examples above do that either.”
Cerebras WSE-3 features 900,000 cores, 44GB SRAM, and 4 trillion transistors
“And so the wafer scale engine three, as I mentioned, 900,000 cores, 44 gigabytes of SRAM, four trillion transistors, and I do add a note here that the paper focuses on wafer scale engine two, and so the wafer scale engine three is, you know, just an upgraded v…”