Ross: GPUs excel at LLM attention while LPUs excel at applying weights
Jonathan Ross · Why Asking the Right Questions Is the Most Important Skill in the AI Age · Jul 5, 2026 · at 27:01
Groq founder Jonathan Ross explains the hybrid architectural division of labor between Groq LPUs and NVIDIA GPUs.
0:00 / 0:19exact quote · 19.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The LPU and GPU, as mentioned, they, they're better at different parts of what's, what's called the decoder layer of an LLM. The GPU is better at the attention portion, and the LPU is better at sort of applying the weights, which is the thing that gets trained as opposed to the memory.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jonathan Ross
Ross: West Coast VCs act like lemmings, East Coast VCs think independently
“Well, the way I like to put it is typical, typical, not all, but typical West Coast VCs are more like Lemmings, and typical East Coast VCs all think that they're smarter than each other.”
Ross: Replace one-on-ones and private emails with open communication to eliminate politics
“And so when you are leading groups of people, if you want to reduce the amount of politics and people going off and, you know, forming side cliques and all this, stop having one-on-ones, have big meetings with everyone who you want to tell something and tell t…”
Ross: Putting excess capital into startups no longer provides an advantage
“What's changed though is that unlike the Keynesian beauty contest where the winner is the one with the most money, in reality, there's a point at which you get enough money and you don't need more. And for the first time in history, the startups are not starve…”
Ross: Splitting LLM pre-fill and generation across different chips is a mistake
“What we realized was, and this is what most people get wrong when they're trying to do this themselves. They'll, they'll take the reading of tokens or what's called pre-fill, and they'll do that on one piece of hardware, and then they'll put the generation of …”
Ross: Faster inference compute directly makes AI models smarter
“Now bring in the LPU, and you can go deeper, faster, you can search faster, and so you can actually make a model smarter by making it faster. And so the realization is, and now that you've got this ability to reflect, think deeply, and change the outcome based…”