Feb 20, 2020 · 20m · mad
Designing an AI Supercomputer // Michael James, Cerebras (FirstMark's Data Driven NYC)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In a presentation at FirstMark's Data Driven NYC, Cerebras co-founder Michael James explains how Cerebras designed and manufactured the Wafer Scale Engine—the world's largest chip—to solve fundamental memory and power bottlenecks in traditional AI computing. He details the hardware-software co-design, key manufacturing breakthroughs, and real-world scientific deployments powering the next generation of artificial intelligence.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 2.9% of the talking time here. How this is scored →
Matt as informed peer 0.0 Guest teaching 2.7 Guest disagreement 1.1 Matt pushing back 0.0
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 2:15 Dismissing conventional advice to defer hardware development
Michael James explicitly rejects the standard industry advice to wait for researchers to settle on algorithms before building hardware, calling it 'the wrong advice' and insisting hardware co-design must happen during rapid change.
Hardest push from Matt ▶ 16:42 Host challenges guest on rival AI startup differentiationMatt Turck directly challenges James by citing specific competitors like Graphcore and SambaNova, asking whether Cerebras is actually doing something distinct or merely pursuing the exact same market opportunity.
Biggest teaching moment ▶ 4:00 Educating audience on von Neumann memory power limitsMichael James walks through a detailed mathematical comparison showing that a traditional von Neumann architecture would require over half a megawatt just for memory transfers to emulate a bird's brain, demonstrating why conventional chips fail for AI workloads.
Matt holds his own ▶ 16:42 Host demonstrates market expertise by naming chip competitorsMatt Turck demonstrates strong industry context by explicitly naming rival AI hardware startups Graphcore and SambaNova to probe Cerebras' precise positioning in the semiconductor landscape.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| The AI Wave & Early Industry Skepticism | 0 | 1 | 1 | 0 | Michael James opens the presentation by describing the surge in AI interest and recounting how early advisors warned that creating dedicated computer hardware for AI was crazy. Because this segment is entirely a guest monologue, host expertise and host pushback are scored zero. | |
| Questioning Conventional Von Neumann Architecture | 0 | 3 | 2 | 0 | James presents a technical breakdown of von Neumann architecture limitations, using a bird brain calculation to show that classical memory access requires excessive power. As a monologue segment, host metrics remain zero while the guest provides architectural context. | |
| A New Architectural Paradigm for Data Science | 0 | 3 | 1 | 0 | James reveals the Cerebras Wafer-Scale Engine, walking the audience through its 462 square centimeter size, 1.2 trillion transistors, and 400,000 cores. The host does not speak in this presentation segment, keeping host-side scores at zero. | |
| Scale Comparison: Cerebras WSE vs. Standard GPU | 0 | 3 | 1 | 0 | The guest explains how Cerebras solved silicon yield problems by routing around micro-defects across 400,000 independent cores instead of discarding whole wafers. Host participation is absent, maintaining zero host scores. | |
| Overcoming Engineering Challenges & Silicon Thermal Expansion | 0 | 4 | 1 | 0 | James details a year-long physical engineering challenge involving thermal expansion and silicon cracking, explaining how electron microscopy uncovered unexpected laser vaporization residue. The host is inactive during this detailed monologue. | |
| Summary of Cerebras WSE Performance Advantages | 0 | 3 | 1 | 0 | James demonstrates how software frameworks map neural network graphs like ResNet-50 onto WSE cores and highlights early adoption by US National Labs. Host metrics are zero as the presentation continues uninterrupted. | |
| Enabling New Categories of Learning Algorithms | 0 | 2 | 1 | 0 | James concludes his talk by encouraging data scientists to break free from GPU-based constraints and invent new categories of learning algorithms. The host does not engage during this closing monologue span. |
Statements from this episode (21)
Cerebras built the first supercomputer designed end-to-end for AI
“And it's also the first supercomputer that has been designed from beginning to end for artificial intelligence and deep learning.”
New computing workloads always demand purpose-built hardware
“When we look throughout the history of computing, it has never happened that a new important workload has come along, and the correct answer to that workload was to use some pre-existing equipment that had been designed to do something else”
Von Neumann architecture might not be the right path for AI
“So, so maybe there's a sign that, that we're not on the right track with the architecture we have in, in von Neumann computers.”
James: Deep learning models rely on data flow, not instruction-driven loops
“The models we're producing don't work based on instructions. It, it's not a central operator that says do this, do that. Instead it's the data that flows over our model and a simple feedback equation that tells that data how to manipulate and transform the mod…”
Cerebras designed a chip as large as possible with interleaved memory
“We decided to build a machine that was as large as we could, that had memory interleaved directly with the computational substrate.”
The Cerebras Wafer Scale Engine is the world's largest processor
“This is the world's largest processor.”
Cerebras's WSE is the first chip with over 1 trillion transistors
“It is the first part that has over a trillion transistors, and we're here, and at 1.2, and it has 400,000 sparse linear algebra cores on it.”
Cerebras's WSE delivers 9 petabytes per second of memory bandwidth
“There's 18 gigabytes of memory right, right on that, ah, on that silicon wafer, and it gives an unprecedented nine gigabytes per second, sorry, nine petabytes per second of memory bandwidth.”
Wafer-scale chips were previously presumed impossible to build
“And when we showed this the world got pretty excited, so there was a large press splash, and in part that's because this had been presumed to be impossible to achieve.”
Cerebras routes around manufacturing defects across its 400,000 cores
“By having 400,000 independent cores, it's perfectly fine if we have a few hundred defects on there, and it's only our job to route the information flows around those defects.”
No motherboard had ever housed a processor as large as Cerebras's
“No one before had ever put a processor that large in a motherboard, and so we had to also make a system to house this thing.”
Cerebras CS-1 system enclosure contains a single wafer plus cooling hardware
“There's only one wafer mounted in there, and the entire rest of this is about, ah, cooling and power delivery.”
Traditional power delivery would carry 40,000 amps and melt Cerebras's processor
“We would have 40,000 amps, and that would actually melt the part if we chose to do it that way, so we had to bring some of the geometric thinking beyond how we think geometry in the models into this three D structure.”
Michael James: Cerebras succeeded at reticle stitching on the first attempt
“Reticle stitching to get above the optical limit of how you manufacture these things that worked the first time we tried it.”
Cerebras's chip provides 10,000x the memory bandwidth of standard GPUs
“So, having solved this, we were left with a machine that was pretty impressive on most of the metrics, being larger, more cores, 10,000 times the memory bandwidth and 33,000 times the fabric bandwidth.”
Programming Cerebras's engine is simpler than programming conventional computers
“So again, the simpler, ah, I think simpler than programming a conventional computer where you have to specify millions of instructions.”
Cerebras maps neural network layers across 400,000 chip cores
“We have, ah, an optimization solver that places these over the 400,000 cores.”
Cerebras's first customer was the U.S. National Labs
“It's pretty unusual in Silicon Valley to have your first customer be the U.S. National Labs”
Lawrence Livermore integrated Cerebras into its Lassen supercomputer
“Lawrence Livermore has integrated into their Lassen supercomputer.”
Cerebras' Wafer Scale Engine is fabricated at TSMC
“It is fabricated at Taiwan Semiconductor, which is one of the world's large major fab houses.”
Quantum and optical computing remain 10 to 15 years away from production
“We think that because that's primary research, that, that's a good, ah, 10 to 15 years out, whereas what we looked to do was to take, ah, pre, ah, technology we could manufacture, maybe difficult, but all achievable, so we weren't trying to invent any new law …”