Opinion
Pope: Groq and Cerebras are uncompetitive on dollars per token
“And then there's the Grok and Cerebris that are much better at latency because they've got this the SRAM, weights are in SRAM very low latency. The problem is, and the challenge when you go to a Grok or a Cerebris system is that the throughput you get there, i…”
Assertion Contradicted
Pope: Google completely stopped publishing its AI research around 2022
“In twenty-twenty-two was about the time when just Google completely stopped publishing its research. And so all the good papers are from before that as a result.”
Prediction Open · timeframe Feb 2029
Pope: Model parameter counts will grow much faster than context lengths
“Really tied into this context thing, I think the context size will stay ballpark the same way it is, maybe a few times larger. But the parameter count will go up. Like, parameter count should grow much, much faster than context length, actually, just because o…”
Insight
Pope: The best AI inference chip is also a great training chip
“I think the best inference chip today will be a train, a really good training chip as well.”
Disclosure
Pope: MatX combines SRAM and HBM for cheap, low-latency AI chips
“It is actually possible to do both in the same chip. You, it's kind of an obvious thing. You say you take the HPM, you take the SRAM, put them together in the same chip, you put the weights in SRAM, and you put the all of the inference data in HPM. That is wha…”
Insight
Pope: Mixture of experts maps well to systolic arrays, attention does not
“The mixture of expert layer maps really well, but the attention does not.”
Disclosure
Pope: MatX splits large systolic arrays without sacrificing efficiency
“Take a really large systolic array, but have a way to split it up into pieces without losing efficiency. So sort of that is the core of the design for us.”
Assertion Partly supported
Pope: Chip tape-outs cost $30M and fail 50% of the time
“The ideal, which companies tend to hit about 50% of the time is that your first tape out costs like thirty million dollars your first... The ideal is that your first tape out is actually, is your production thing. So you do a tape out, you make maybe a thousan…”
Assertion Supported
Pope: OpenAI is starting to design its own custom chips
“Google does. OpenAI is starting.”
Assertion Not checkable as stated
Pope: AI labs apply RL to software code, not chip architecture
“Models are extremely good at Rust and Python. They've done a lot of RL on them. They have not done as much RL on Verilog. They've done almost none on, okay, write, write me a markdown file that describes a chip architecture.”
Assertion Not checkable as stated
Pope: Almost every senior AI researcher passed through Google Brain
“Pretty much anyone who's maybe, I don't know, over 30 and at a large lab has been at Google Brain at some point.”
Opinion
Pope: Google's decision to build TPUs specifically for neural nets paid off
“They at least had the option, like the opportunity to design the TPUs for neural nets at least rather than graphics applications like NVIDIA. And so the overall architecture starting with single core doing what was at the time reasonably large systolic arrays …”
Opinion
Pope: TPUv1 announcement catalyzed the 2016-2017 AI chip startup wave
“TPUv one was announced in 2016, I think. That was what actually kind of led to the creation of all of those 2016, 20 17 startups. So Cerebus, Gronk, Graphcore, SambaNova, all of those.”
Prediction Open · timeframe Feb 2029
Pope: MatX will sell AI inference chips first due to lower risk
“Our product is both training and inference, but I think the first sales will be an inference. That's mostly just a market effect where It's easier to buy, like, it's not as big of a risk to go to buy an inference cluster than as a training cluster.”
Disclosure
Pope: MatX raised a $500M Series B co-led by Jane Street
“So we this is a we've raised a series B round. It's led by Jane Street and Situational Awareness Situational awareness, that is Leopold Ashenbrenner's fund. He wrote the definitive book on, on, on where, on AGI and where it's going. And then Jane Street, they'…”
Prediction Not checkable as stated
Pope predicts supply crunches across the entire AI hardware supply chain
“The supply chain we're gonna have crunches on, on all of the supply chain, really. So if you look at the, sort of, the big components of what any company, but like us, for example, build-out there is dependency on Logic Dies from typically TSMC, maybe Samsung,…”
Insight
Pope: Large systolic arrays are unbeatable in area and power efficiency
“Make a really large systolic array. You can't beat that in area or power efficiency.”
Prediction Not checkable as stated
Pope: 4-bit precision will probably be the primary AI format, like Nvidia
“We think probably the main thing will be similar to where NVIDIA is at, which is four bit precision.”
Disclosure
Pope: MatX trains small LLMs from scratch daily for hardware co-design
“So our ML team is actual, real ML research. What they do every day is they train small LLMs from scratch, focusing on numerics and attention.”
Assertion Supported
Pope: Google built TPUv1 in 12-18 months with 20-30 engineers
“TPUv one actually was, I think, is a really impressive project. It was done on a very short timeline, maybe, I don't know the full details, but maybe about a year or so, maybe a year and a half with a skeleton team of 20, 30 people.”
Insight
Pope: Chip physics enforces parallelism because signals take 100 cycles to cross
“So, I mean, it is just true hardware is massively parallel. Like, you've got tens of billions, hundreds of billions of transistors on your chip, and it takes, like, maybe a hundred clock cycles to get from one side of the chip to the other, and so you can't, l…”
Insight
Pope: Instruction control dominates CPU cost, unlike GPUs with large payloads
“Reading what do I have to do next? Okay, how do I do that? That is most of the cost on a CPU, whereas if you just keep the same instructions but make the payload a hundred times bigger, then you can shift most of the cost to be in the actual work that you want…”
Assertion Supported
Pope: SRAM is an order of magnitude faster per token than HBM
“There's, yeah, there's just some simple math of, like, how long does it take you to read through all of HBM? It takes about 20 milliseconds, and so that's the amount of time per token it runs. Yes. Whereas the amount of time to read through all of SRAM is much…”
Assertion Supported
Pope: Metal layer chip respins cost $100K versus $30M full tape-outs
“In good cases and in many cases, you can redo just the metal layers, which costs you only like a 100,000 dollars. As opposed to the- Pay the thirty million dollars again. But in bad cases, like, if you've made something serious and you can't fix that at the me…”
Reiner Pope