Feb 26, 2026 · 1h 13m · cheeky-pint
Reiner Pope of MatX on accelerating AI with transformer-optimized chips
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Former Google TPU architect and MatX co-founder Reiner Pope breaks down the history, architectural trade-offs, and future of specialized AI silicon, highlighting how tailored hardware and software co-design are essential for scaling modern large language models.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. John holds 29.2% of the talking time here. How this is scored →
John as informed peer 4.7 Guest teaching 5.8 Guest disagreement 1.4 John pushing back 1.8
speaking balance: gold is John, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 32:43 Software companies ship bugs too
Pope dryly dismisses Collison's suggestion that thirty million dollar tape-outs shouldn't have logical bugs by comparing it directly to software companies shipping bugs to production.
Hardest push from John ▶ 38:03 Pushing on why TSMC faces no competitorsCollison repeatedly challenges the assumption that TSMC's position is inevitable, pointing out that ship-building and airplane manufacturing have competitive markets.
Biggest teaching moment ▶ 15:22 Explaining the SRAM vs HBM memory tradeoffPope gives a detailed technical breakdown of why Grok and Cerebras struggle with throughput despite low latency, explaining how combining SRAM and HBM solves both.
John holds their own ▶ 1:01:00 Connecting gate heuristics to Jeff DeanCollison demonstrates deep familiarity with high-scale systems folklore by connecting Pope's internal 'go/gates' rules to Jeff Dean's classic systems performance numbers.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | John as informed peer | Guest teaching | Guest disagreement | John pushing back | Why |
|---|---|---|---|---|---|---|
| Origins of TPU and Hardware Parallelization | 5 | 5 | 2 | 2 | Collison explores the origins of TPU and asks whether the timing of TPUs and transformers was coincidental or rooted in parallelization insights. Pope educates Collison on the mechanical sympathy required for parallel hardware versus CPU instruction parallelism. | |
| Comparing CPUs and GPUs for AI Workloads | 4 | 6 | 1 | 1 | Collison asks for an intuitive mental model comparing CPUs and GPUs beyond the tautology that GPUs do math. Pope provides a clear analogy comparing a truck carrying a large payload to a motorcycle navigating an obstacle course. | |
| Founding MatX and Core Value Proposition | 5 | 5 | 2 | 2 | Pope outlines MatX's founding thesis around matrix sizes and lower precision arithmetic. Collison pushes on whether product risk exists given that LLMs are clearly established, prompting Pope to clarify that the bet was made prior to ChatGPT. | |
| Scaling MatX, Funding, and AI Supply Chain Bottlenecks | 5 | 6 | 1 | 2 | Collison asks how MatX will compete with giants for supply chain allocations across wafers, HBM, and racks. Pope breaks down the exact trade-offs between SRAM and HBM memory latencies and explains how ironclad buyer contracts secure supplier capacity. | |
| MatX Architecture: Memory, Systolic Arrays, and Low Precision | 4 | 7 | 1 | 1 | Pope explains MatX's architecture combining SRAM, HBM, splittable systolic arrays, and 4-bit precision. Collison asks basic clarifying questions on bit precision and systolic arrays while Pope explains the mathematical trade-offs. | |
| The Chip Design Process: From Verilog to Tape Out | 4 | 7 | 2 | 2 | Collison explores the physical chip design cycle, comparing software iteration to chip tape-outs. When Collison asks why simulation doesn't eliminate all errors before spending thirty million dollars, Pope retorts that software companies also ship bugs to production. | |
| Software Moats, TSMC Dominance, and Space Data Centers | 6 | 5 | 2 | 4 | Collison pushes Pope on why TSMC has maintained an uncontested monopoly at leading edge nodes despite geopolitical risks, and presses on the viability of space data centers. Pope counters that leading-edge nodes only matter for specific power-sensitive workloads and details chip redundancy models in space. | |
| Sponsor Segment: Stripe Billing for AI Products | 4 | 5 | 1 | 2 | Segment starts with Collison reading a Stripe Billing ad read before discussing AI development bottlenecks and state management in context windows. Pope explains how context size struggles to grow while parameter counts expand. | |
| Team Culture, Numerics Research, and In-Head Iteration | 5 | 6 | 1 | 1 | Pope outlines MatX's unique ML research team co-designing numerics and doing rapid in-head estimation before simulation. Collison connects Pope's gate-cost heuristic to Jeff Dean's famous 'numbers every engineer should know'. | |
| Why Rust, Systems Optimization, Cuckoo Hashing, and Architecture Frontiers | 5 | 6 | 1 | 1 | Pope explains his preference for Rust over Go and Haskell due to low-level memory control and discusses optimizing SIMD vector instructions for cuckoo hashing. Collison probes into practical workloads and wraps up on architectural opportunities. |
Statements from this episode (26)
Pope: Almost every senior AI researcher passed through Google Brain
“Pretty much anyone who's maybe, I don't know, over 30 and at a large lab has been at Google Brain at some point.”
Pope: Google's decision to build TPUs specifically for neural nets paid off
“They at least had the option, like the opportunity to design the TPUs for neural nets at least rather than graphics applications like NVIDIA. And so the overall architecture starting with single core doing what was at the time reasonably large systolic arrays …”
Pope: TPUv1 announcement catalyzed the 2016-2017 AI chip startup wave
“TPUv one was announced in 2016, I think. That was what actually kind of led to the creation of all of those 2016, 20 17 startups. So Cerebus, Gronk, Graphcore, SambaNova, all of those.”
Pope: Google built TPUv1 in 12-18 months with 20-30 engineers
“TPUv one actually was, I think, is a really impressive project. It was done on a very short timeline, maybe, I don't know the full details, but maybe about a year or so, maybe a year and a half with a skeleton team of 20, 30 people.”
Pope: Google completely stopped publishing its AI research around 2022
“In twenty-twenty-two was about the time when just Google completely stopped publishing its research. And so all the good papers are from before that as a result.”
Pope: Chip physics enforces parallelism because signals take 100 cycles to cross
“So, I mean, it is just true hardware is massively parallel. Like, you've got tens of billions, hundreds of billions of transistors on your chip, and it takes, like, maybe a hundred clock cycles to get from one side of the chip to the other, and so you can't, l…”
Pope: Instruction control dominates CPU cost, unlike GPUs with large payloads
“Reading what do I have to do next? Okay, how do I do that? That is most of the cost on a CPU, whereas if you just keep the same instructions but make the payload a hundred times bigger, then you can shift most of the cost to be in the actual work that you want…”
Pope: The best AI inference chip is also a great training chip
“I think the best inference chip today will be a train, a really good training chip as well.”
Pope: MatX will sell AI inference chips first due to lower risk
“Our product is both training and inference, but I think the first sales will be an inference. That's mostly just a market effect where It's easier to buy, like, it's not as big of a risk to go to buy an inference cluster than as a training cluster.”
Pope: MatX raised a $500M Series B co-led by Jane Street
“So we this is a we've raised a series B round. It's led by Jane Street and Situational Awareness Situational awareness, that is Leopold Ashenbrenner's fund. He wrote the definitive book on, on, on where, on AGI and where it's going. And then Jane Street, they'…”
Pope: Groq and Cerebras are uncompetitive on dollars per token
“And then there's the Grok and Cerebris that are much better at latency because they've got this the SRAM, weights are in SRAM very low latency. The problem is, and the challenge when you go to a Grok or a Cerebris system is that the throughput you get there, i…”
Pope: MatX combines SRAM and HBM for cheap, low-latency AI chips
“It is actually possible to do both in the same chip. You, it's kind of an obvious thing. You say you take the HPM, you take the SRAM, put them together in the same chip, you put the weights in SRAM, and you put the all of the inference data in HPM. That is wha…”
Pope: SRAM is an order of magnitude faster per token than HBM
“There's, yeah, there's just some simple math of, like, how long does it take you to read through all of HBM? It takes about 20 milliseconds, and so that's the amount of time per token it runs. Yes. Whereas the amount of time to read through all of SRAM is much…”
Pope predicts supply crunches across the entire AI hardware supply chain
“The supply chain we're gonna have crunches on, on all of the supply chain, really. So if you look at the, sort of, the big components of what any company, but like us, for example, build-out there is dependency on Logic Dies from typically TSMC, maybe Samsung,…”
Pope: Large systolic arrays are unbeatable in area and power efficiency
“Make a really large systolic array. You can't beat that in area or power efficiency.”
Pope: Mixture of experts maps well to systolic arrays, attention does not
“The mixture of expert layer maps really well, but the attention does not.”
Pope: MatX splits large systolic arrays without sacrificing efficiency
“Take a really large systolic array, but have a way to split it up into pieces without losing efficiency. So sort of that is the core of the design for us.”
Pope: 4-bit precision will probably be the primary AI format, like Nvidia
“We think probably the main thing will be similar to where NVIDIA is at, which is four bit precision.”
Pope: Chip tape-outs cost $30M and fail 50% of the time
“The ideal, which companies tend to hit about 50% of the time is that your first tape out costs like thirty million dollars your first... The ideal is that your first tape out is actually, is your production thing. So you do a tape out, you make maybe a thousan…”
Pope: Metal layer chip respins cost $100K versus $30M full tape-outs
“In good cases and in many cases, you can redo just the metal layers, which costs you only like a 100,000 dollars. As opposed to the- Pay the thirty million dollars again. But in bad cases, like, if you've made something serious and you can't fix that at the me…”
Pope: NVIDIA includes eight spare chips in a 64-chip rack
“Nvidia has eight spare chips and a rack of 64.”
Pope: AI labs apply RL to software code, not chip architecture
“Models are extremely good at Rust and Python. They've done a lot of RL on them. They have not done as much RL on Verilog. They've done almost none on, okay, write, write me a markdown file that describes a chip architecture.”
Pope: Foundry turnaround from tape-out to first chips is 4-5 months
“What we see is time from, like, tape out to first chip, to chips back. Again, depends on node, but it's ballpark four or five months.”
Pope: Model parameter counts will grow much faster than context lengths
“Really tied into this context thing, I think the context size will stay ballpark the same way it is, maybe a few times larger. But the parameter count will go up. Like, parameter count should grow much, much faster than context length, actually, just because o…”