Feb 17, 2025 · 1h 25m · news
Jonathan Ross, Founder & CEO @ Groq: NVIDIA vs Groq - The Future of Training vs Inference | E1260 · 20VC with Harry Stebbings
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this comprehensive interview, Groq Founder and CEO Jonathan Ross discusses the critical distinctions between AI training and inference, highlighting how Groq's innovative chip architecture and unique non-dilutive business models position the company to dominate the inference market while coexisting with NVIDIA.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Harry holds 13.6% of the talking time here. How this is scored →
Harry as informed peer 4.6 Guest teaching 5.3 Guest disagreement 2.4 Harry pushing back 4.2
speaking balance: gold is Harry, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:05:53 Slothful Incumbents Rejection
Jonathan aggressively rejects Harry's concern about hurting traditional companies, declaring there is no right to protect slothful incumbents who fail to adapt to technological disruption.
Hardest push from Harry ▶ 30:53 Grilling Groq on NVIDIA Margin UndercutHarry forcefully challenges Jonathan's positioning by pointing out that NVIDIA's massive 80% gross margins give them the financial firepower to slash prices and destroy Groq's business model.
Biggest teaching moment ▶ 4:07 Big O Complexity and Sorting AlgorithmsJonathan takes Harry to school on fundamental computer science, explaining Big O complexity, bubble sort versus quicksort, and why LLMs face strict mathematical boundaries when performing arithmetic.
Harry holds his own ▶ 50:31 Silicon Valley Perks vs DeepSeek RealityHarry demonstrates sharp domain knowledge by contrasting Silicon Valley's inflated $2M salaries and cushy perk environment with Chinese engineers at DeepSeek working 20-hour days in Guangdong.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Harry as informed peer | Guest teaching | Guest disagreement | Harry pushing back | Why |
|---|---|---|---|---|---|---|
| Episode Preview and High-Impact Hooks | 2 | 4 | 1 | 1 | Harry welcomes Jonathan and opens with broad introductory questions about scaling laws and DeepSeek. Jonathan politely reframes how synthetic data changes logarithmic scaling curves. | |
| The Mathematics of LLMs and Reasoning Limits | 2 | 7 | 2 | 3 | Jonathan delivers a computer science lesson on Big O complexity, contrasting quicksort and bubble sort to show why LLMs face mathematical limits when multiplying large numbers. Harry asks standard clarifying questions about efficiency ceilings. | |
| Fast vs. Slow Thinking and Test-Time Compute | 3 | 5 | 1 | 3 | Jonathan connects Daniel Kahneman's fast vs. slow thinking framework to test-time compute. Harry probes on whether hardware, energy, or algorithms represent the primary system bottleneck. | |
| DeepSeek's Impact and Soft vs. Hard Bottlenecks | 5 | 5 | 3 | 5 | Harry challenges Jonathan using DeepSeek's efficiency breakthrough to question the necessity of massive compute. Jonathan corrects the premise, explaining that DeepSeek represented an algorithmic improvement that enabled easier synthetic data generation. | |
| Timing the Wave: The Smartphone and Uber Analogy | 2 | 3 | 1 | 2 | Jonathan shares an extended narrative about timing market waves and Groq's near-bankruptcy, where employees took pay cuts via Groq Bonds. Harry acts primarily as an empathetic listener. | |
| The LPU Paradigm Shift: Compute as an Employee | 4 | 5 | 1 | 2 | Harry references Hamilton Helmer's 7 Powers framework to ask about chip supply constraints. Jonathan explains HBM memory supply bottlenecks and NVIDIA's monopsony position. | |
| Groq's Architectural Solution: The Pipeline Assembly Line | 3 | 6 | 2 | 3 | Jonathan outlines Groq's LPU architecture, using a moped vs. freight train analogy to explain data center energy efficiency. Harry questions the counter-intuitive physics of using more chips to save energy. | |
| Coexistence and the Nitro Boost Strategy | 4 | 4 | 1 | 3 | Harry notes that enterprise customers order GPUs a year in advance, asking how Groq deploys faster. Jonathan highlights Groq's 51-day deployment timeline in Saudi Arabia due to simplified networking. | |
| Specsmanship vs. Real Value in Enterprise Sales | 5 | 5 | 4 | 5 | Harry presses Jonathan on why NVIDIA does not market LPUs to protect shareholder value. Jonathan rejects the premise and criticizes enterprise specsmanship and vanity metrics. | |
| No Direct Competition: NVIDIA vs. Groq | 7 | 5 | 4 | 7 | Harry forcefully pushes back on Jonathan's claim that NVIDIA isn't a competitor, pointing out NVIDIA's clear desire to dominate inference. Jonathan reframes the market dynamic as complementary. | |
| Hardware Margins and Financial Partnership Models | 7 | 5 | 4 | 7 | Harry grills Jonathan on margin structures, arguing NVIDIA's 80% margins allow them to cut prices and destroy competitors. Jonathan details Groq's non-CapEx partner finance model. | |
| The Data Center Power Squeeze & The Echo Chamber | 5 | 7 | 3 | 5 | Jonathan reveals severe infrastructure bottlenecks like 90-month generator lead times and misinformed real-estate developers. Harry probes the gap between expanding inference demand and prospective data center oversupply. | |
| The Aramco Deal Structure and Financial Viability | 6 | 5 | 3 | 5 | Harry demonstrates strong financial grasp by pointing out that Groq's Aramco deal was $1.5B in revenue rather than venture capital funding. Jonathan explains positive contribution margins and sub-exponential growth. | |
| Hyperscaler Dynamics and Market Caps | 6 | 5 | 2 | 4 | Harry cites specific CapEx figures for Meta, Microsoft, and Google. Jonathan breaks down the three stages of startup maturity and the disruption cycles affecting incumbents. | |
| Talent Wars and Compensation Realities | 6 | 6 | 3 | 6 | Harry offers a sharp comparison between Silicon Valley's $2M salaries and perk-heavy culture versus DeepSeek workers in Guangdong grinding 20-hour days. Jonathan introduces the Keynesian Beauty Contest concept. | |
| Data Privacy and Groq's Sub-Linear Scaling Strategy | 4 | 7 | 3 | 4 | Jonathan explains Groq's sub-linear management philosophy, showing how 300 employees built full-stack chips and software by applying Big O complexity to org design. Harry asks how team scaling limits are managed. | |
| Geopolitics: China vs. US vs. Europe in AI | 6 | 6 | 3 | 5 | Harry asks whether China actually lacks Blackwell chips given regional pass-throughs. Jonathan argues CCP political censorship and fear of executive failure represent the true ceiling on Chinese AI innovation. | |
| Regulatory Ideals and "City F" for Europe | 6 | 6 | 4 | 7 | Harry notes Europe hired 1,500 AI regulators and challenges Jonathan's regulatory-free City F proposal as unfair to established corporations. Jonathan forcefully asserts that slothful incumbents deserve no protection. | |
| Personal Motivations, Loss Bias, and Quick-Fire Predictions | 5 | 6 | 2 | 5 | Jonathan shares a personal childhood memory of his father losing fortunes to illustrate loss bias in hiring. Harry shares details of his team's AI research workflow and probes human complacency. | |
| Quick Fire: Delegation vs. Founder Mode and Groq's Challenge Coin | 5 | 5 | 4 | 5 | Jonathan rejects Founder Mode as a sign of poor delegation and presents Groq's physical alignment challenge coin. He also offers colorful commentary on Sam Altman, JD Vance, and Elon Musk in Paris. | |
| Scaling Chip Production Without the Fear of Failure | 4 | 5 | 2 | 4 | Jonathan shares his 70 lb weight loss on GLP-1 medication to predict a sudden technological breakthrough in slowing human aging. Harry agrees based on medical research trends. | |
| Finding Product-Market Fit and the Three Types of Happiness | 4 | 6 | 2 | 2 | Jonathan defines Type 3 future happiness unique to pre-PMF founders and outlines four technical milestones required before reaching true generative AI autonomy. | |
| Evaluating AI Agent Startups and Groq's Rising Revenue | 4 | 5 | 1 | 4 | Harry shares personal motivation regarding MS research for his mother. Jonathan envisions natural language prompt engineering unlocking 1.4 billion African entrepreneurs. |
Statements from this episode (61)
Ross: Groq benefits NVIDIA by capturing lower-margin inference workload
“You can almost say we're one of the best things that's ever happened to NVIDIA because they can make every single GPU that they were going to make, and they can sell it for training, high margin, gets amortized across deployment. You know, we'll take the low m…”
Ross: Relevance and market foothold outweigh profitability during hypergrowth
“And when you are growing faster than exponential, there's no amount of profit that you can make that matters. What matters is getting a toehold in the market and becoming relevant.”
Ross: Scaling laws are misunderstood because data quality varies
“They're misunderstood because the assumption is that all of the data is the same quality.”
Ross: LLM-generated synthetic data is better for model training
“You could have an LLM generate synthetic data, and when it generates the synthetic data, the data is better. You then train on that synthetic data.”
Jonathan Ross: LLMs cannot multiply large numbers without intermediate reasoning steps
“One of the reasons that these LLMs struggle to multiply large numbers is because multiplying is not linear. These LLMs could do anything linear without, you know, needing to think, but just like on a piece of paper how you need to write out all those intermedi…”
Jonathan Ross: Training AI for intuitive reasoning requires 10x more data
“If your job is to get better at multiplying numbers, and I tell you that I want you to be able to do it with fewer steps, more intuitively, for you to be able to multiply three-digit numbers versus two-digit, you need 10 x the data, and you need 10 x the examp…”
Jonathan Ross: AI scaling is bottlenecked by compute, data, and algorithms simultaneously
“It is the compute. It is the data. It is the algorithms. It's all three of them.”
Ross: Compute is a soft AI bottleneck that compensates for weak data
“Compute has been more of a less of a bottleneck and more of a sort of a, you know, soft neck or something, right? Where, when you provide even more compute, you can sort of overpower the lack of data, the lack of improvement in algorithms. So it's not a hard b…”
Ross: DeepSeek's breakthrough was an algorithmic gain in data generation
“There was an algorithmic improvement on that. And the algorithmic improvement, as I explained, you know, is this seemingly silly thing where they just wrote the answer in a box and then they knew what to look for rather than having to have a human being check …”
Ross: The AI industry mistakenly believed training was costlier than inference
“When we started, the first misconception, which people don't hold anymore, is that training was more expensive than inference.”
Ross: Drop in NVIDIA stock was a misunderstanding of market fundamentals
“I don't agree that Nvidia stock should have gone down for that. I think that was a misunderstanding on most people's part, but it also shows, I think that shows more like everyone keeps saying Nvidia stock can't possibly go higher.”
Groq operated for seven years before achieving product-market fit
“We were around for seven years before we had product market fit, right?”
80% of Groq Employees Accepted Equity Swaps During Cash Crunch
“Instead of leaving, about 80% of the employees participated, 50%, I think, went to the statutory minimum salary by law.”
Jonathan Ross: Groq scaled from 640 to over 40,000 chips in 2024
“We started 2024 with about 640 chips in production. We ended with over 40,000.”
Jonathan Ross: Groq aims to reach over 2 million chips in 2025
“This year we want to be at over two million, and next year the number is much, much, much larger.”
Ross: Groq will need nearly entire fab capacity next year
“For us to hit our numbers next year, which I'm not sharing publicly, we're going to gonna need almost all of the capacity of the fab that we're using.”
Ross: NVIDIA holds a cornered resource as HBM monopsony buyer
“You don't normally think of tech companies as having a cornered resource, but Nvidia has a cornered resource. They're a monopsony, the opposite of a monopoly, a single buyer for HBM, and the Interposer, the COOS.”
AI performance scaled at 4x every 18-24 months via chip accumulation
“Turns out, the number of chips was also doubling every 18 to 24 months. So rather than two X, it was four X.”
Groq deploys 600 to 3,000 chips per AI model instead of eight
“So rather than using eight chips, we'll use 600 or 3000 for a model.”
Ross: Groq's chip architecture improves energy efficiency 3x per token
“It improves at about three X, and the reason is...”
Ross: Edge computing is less energy efficient than data center compute
“They think that edge computing is lower energy. Actually, edge computing is less energy efficient than computing in the data center.”
Ross: Inference Accounts For Roughly 40% Of Nvidia's Market
“Right now, about 40% of their, you know, market is inference.”
Ross: Deploying Cheaper Inference Chips Will Fuel More AI Training
“I think if we were to deploy a lot of much lower cost inference chips what you would see is that same number of GPUs would be sold, but the demand for training would increase because the more inference you have, the more training you need, and vice versa.”
Ross: Some GPU Customers Wait Over a Year After Paying
“Actually, we've spoken with some customers that put orders in over a year in advance. They paid a year in advance and still haven't gotten them.”
Groq Completed Saudi AI Deployment in 51 Days
“Ah, the recent deployment we did in, in Saudi Arabia 51 days from contract to the first tokens being served in production in country.”
Groq Chips Function as Switches Without External Hardware
“We actually don't use switches to communicate between our chips. We just plug our chips into our chips. Our chips are the switch.”
Ross: NVIDIA used misleading benchmark curves to claim 30x GPU speedup
“Like, if you look at the last GTC, there was an announcement that the latest GPUs were 30 x faster than the previous generation. And when you look at how it was done, there was this curve that looked kind of like this, and then it Basically ended here, and the…”
Ross: Tokens per dollar and per watt are the only AI chip metrics that matter
“Like, just tell me what the tokens per dollar is, and tell me what the tokens per watt is. Nothing else really matters.”
Jonathan Ross: Competing directly means a startup failed to find unsolved problems
“If you're competing, it means that you haven't found an unsolved customer problem, because if you're competing, someone else has already solved the problem.”
Ross: Customers buy both Groq and NVIDIA GPUs rather than replacing Groq
“We don't really have people saying, you know, we're gonna buy GPUs instead of you. We do have people saying we're gonna buy both.”
Ross: Groq LPUs Cost 5x Less Than Latest GPUs for Inference
“More than five x lower. Just the memory alone in the latest GPUs costs more than our fully loaded CapEx per chip deployed.”
Ross: GPU Operational Cost Alone Equals Groq Total CapEx plus OpEx
“So we use about a third of the energy per token. About over a three-year period, one-third of our cost is the OpEx, which is mostly energy and data center rent, and two-thirds is the CapEx, which means that since we're one-third of the energy, the cost to run …”
Ross: Groq earns ~20% upfront margins on chip deployments
“Anywhere from, depending on the deal we do get some on the back side, but up front it's about 20%.”
Ross: Partners fund Groq's CapEx until target IRR is achieved
“So the deals that we do the partner will off, because we don't deploy, we don't spend money for our own capex. The partner will put up the money for us to deploy. We pay back with a, you know, decent IRR, and, but we split, and most of it goes to the partner, …”
Ross: Worldwide data center capacity is currently 15 gigawatts
“I am aware of about 20 gigawatts of power that people want to make available for data centers now. Right now there's about 15 gigawatts of data centers worldwide, so more than double the current capacity.”
Ross: Power will become a hard bottleneck for AI in 3-4 years
“That power will become a hard bottleneck in three to four years.”
Ross: Industrial generators currently face a 90-month lead time
“There's a 90 month lead time on generators right now”
Ross: Most newly announced speculative data center projects will never be built
“Most of these projects will never be developed.”
Ross: Groq is sole profit-making provider running open-source AI models
“But actually we have a very positive contribution margin right now. And so as far as we know, we're the only ones that are actually making money running these open source models”
Ross: Groq plans to serve 50% of global AI inference by 2027
“And our goal by the end of 20, 27 is to be providing at least half of the world's AI inference compute.”
Ross: LLMs are better than search, forcing Google to reinvent itself
“Now, Google, ah, has to redo this because LLMs are better than search, right?”
Ross: Total AI Industry Returns Will Exceed Capital Invested Despite Mass Losses
“I can guarantee you that a huge amount of money will be incinerated. But I also bet that in total more money will be made than will be put in.”
Groq maintains policy never to offer highest salary in bidding wars
“We have a policy that we never offer the highest because we want people to choose us, not choose the salary.”
Ross: Groq will not train proprietary AI models to avoid competing with clients
“We have decided that we're not going to train our own models. We'll do a little fine tuning for specific cases or whatnot, but we don't want to compete.”
Ross: Groq built its entire hardware and cloud stack with 300 people
“We have 300 people. We built our own chip. We built our own networking hardware and software. We built our own runtime. We built our own orchestration layer. We built our own compiler. We built our own cloud. We built all this with 300 people.”
Ross: Accepting B-players as a company grows reflects management laziness
“I think saying that you're gonna hire B players because you've gotten large enough is, is laziness and an excuse. And it, it's a lack of creativity in your business model and how you're going, the algorithm of how you're going to scale.”
Ross: Global power limits will block Chinese AI chip exports
“If they want to go out into the world and deploy chips like they did with Huawei and networking gear, that's going to be complicated because people aren't going to have the power around the world to run more expensive accelerators.”
Ross: Southeast Asian GPU deployments secretly serve Chinese firms
“One of the concerns right now is about Malaysia or Singapore, that region over there being a place where people are deploying GPUs with the wink, wink, like we're not going to rent it to China, right? But that's a belief that a lot of people are doing that. Ot…”
Stebbings: EU has hired 1,500 people for AI safety and policing
“The EU has supposedly hired 1500 people for AI safety and policing.”
Ross: European Notice Periods and Hiring Rules Suppress Employee Wages
“If you are a company right now, it feels like, okay, well, it's harder to poach, but what does that do? It suppresses wages. It's harder to hire someone. They're less likely to move. There's less competition. It suppresses wages.”
Ross: AI will exacerbate society's loss of drive and agency
“And so we have this sort of financial diabetes as a society, and I think it's going to get worse with AI.”
Ross opposes 'founder mode', advocating delegation over micromanagement
“I'm anti-founder mode. I believe in delegation. I think when you are telling people how to do their job, that is an indication that it's not a, not necessarily a problem with you. It could just be that that person is not right for that job.”
Groq gives every employee a 25 million token per second challenge coin
“Everyone at Grok carries this twenty five million token per second challenge coin. And what this is, is it tells everyone what we're doing.”
Ross: Musk's OpenAI offer was driven by jealousy of Altman
“Frankly, I think I think Elon was a little jealous that Sam Altman was sitting next to JD Vance, and it wasn't him. And because it was right around the time that Sam Altman was speaking that he announced it.”
Ross: I would have mocked Musk's OpenAI bid using a Twitter $420 joke
“I would have probably said instead of whatever he said about nine billion, I would have said, yeah, I'm going to take Twitter public at 420 dollars a share.”
Ross: AI could deliver longevity breakthrough within 10 years
“Belief is that if it is possible to significantly slow or stop aging, I think that you will have a Manjaro moment in maybe the next 10 years.”
Ross: Founders live entirely on future happiness prior to product-market fit
“As a founder, the only type of happiness you get is this third type, which is future happiness. The other two, the common ones are the present is happy, right? And the other one is you went through some real crappy stuff, but the memories are, make you happy, …”
Ross: Effective agentic AI requires solving hallucinations first
“I think agentic comes after you solve the hallucination problem because otherwise you've got these long chains where you can introduce hallucinations. It'll kind of work, but it'll work much better after.”
Ross: LLM writing is poor because language models are predictable
“The reason that the writing from LLMs is terrible is because it's predictable.”
Ross: Prompt engineering will unlock 1.4 billion potential African entrepreneurs
“I think prompt engineering is going to unlock a huge swath of human society. There's 1.4000000000 people in Africa who know how to speak, and if you were to give them access to a tool that They could create applications live just by speaking to it. That would …”
Shorts cut from this episode
▶ The Data Center Problem No One is Talking About · 20VC with (@34:32)
▶ How Nvidia and Groq can Win Together · 20VC with Harry Stebb (@0:11)
▶ The problem with data centres 🔋 · 20VC with Harry Stebbings (@34:32)
▶ How ‘Groq Bonds’ saved them from running out of money 💸 · 2 (@13:12)
▶ Elon Musk vs. Sam Altman beef 🥊 · 20VC with Harry Stebbings (@1:14:58)
▶ The 4 future opportunities in AI 🤖 · 20VC with Harry Stebbi (@1:20:35)
▶ Does China have an advantage in AI? · 20VC with Harry Stebbi (@58:43)
▶ How NVIDIA and Groq can dominate AI together? · 20VC with H (@0:11)
▶ Is this the best thing for NVIDIA? · 20VC with Harry Stebbi (@0:00)