Feb 19, 2025 · 1h 8m · tbpn
How Does GROK Compare? (Full Analysis)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
John Coogan and his co-host deliver a comprehensive analysis of xAI's Grok 3 launch, evaluating its technical benchmarks, compute scaling limits, and infrastructure footprint. The discussion broadens to explore foundation model economics, startup strategies like Ilya Sutskever's SSI, and the escalating personal rivalry between Elon Musk and Sam Altman.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 72.4% of the talking time here. How this is scored →
The hosts as informed peer 6.2 Guest teaching 1.2 Guest disagreement 1.1 The hosts pushing back 1.4
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 7:11 Pushing back on cluster size definition
The co-host challenges whether the stated GPU figures referred to xAI's internal cluster or any frontier model historically.
Hardest push from the hosts ▶ 4:20 Host directly refuting cluster claimCoogan explicitly counters the co-host by stating 'That is not true' regarding the claim that 20,000 GPUs was the largest cluster ever built.
Biggest teaching moment ▶ 1:05:30 Altman's public psychology diagnosisAltman's cited commentary reframes Musk's aggressive hostile bidding as driven by lifelong personal insecurity rather than purely strategic logic.
The host holds their own ▶ 40:20 Explaining FLOPs order of magnitudeCoogan demonstrates strong domain expertise by detailing why GPU counts alone are insufficient and explaining FLOP notation from E23 to E26.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Grok 3 Launch and Stratechery Benchmark Analysis | 6 | 1 | 1 | 2 | John Coogan initiates the breakdown of Grok 3 benchmarks from Stratechery, elaborating on compute trade-offs between O3 variations and inference costs. | |
| Andrej Karpathy's Grok 3 Takeaways and xAI's Rapid Scaling | 6 | 2 | 2 | 4 | Coogan corrects the co-host on GPU cluster sizes, clarifying that xAI used 20k GPUs for training Grok 3 rather than the full 100k Memphis cluster. | |
| Distribution Advantages, Monetization, and the 1 Million GPU Cluster | 6 | 1 | 2 | 1 | Coogan analyzes OpenAI's need for ad infrastructure and Microsoft partnerships while discussing xAI's 1-million GPU ambition. | |
| Ilya Sutskever's SSI and the $30 Billion Valuation | 5 | 2 | 1 | 1 | Both hosts explore SSI's 30 billion dollar valuation and share perspectives on founder succession and startup acquisitions. | |
| Foundation Model Wars and the Trillion-Dollar Commodity Thesis | 7 | 1 | 1 | 1 | Coogan articulates a macro thesis that intelligence behaves like high-value commodities such as oil, comparing market capitalizations across major tech waves. | |
| AI Moats, Solo Builders, and Physical Infrastructure Requirements | 6 | 1 | 1 | 1 | Coogan reviews Yassine's contrarian post on software moats, discussing physical hardware moats and solo builder economics. | |
| Timeline Reactions, Benchmark Omissions, and the Memphis Colossus Buildout | 7 | 1 | 1 | 1 | Coogan explains power fluctuation mitigation in data centers and details xAI's rapid Colossus facility buildout in Memphis. | |
| Model Fine-Tuning Echo Chambers and the Revival of the Bitter Lesson | 6 | 1 | 1 | 1 | Coogan notes user timeline personalization leading to confirmation bias echo chambers and references Sutton's Bitter Lesson. | |
| Karpathy's Hands-On Evals: Logic, Deep Search, and AI Humor | 6 | 1 | 1 | 1 | Coogan examines Karpathy's prompt evaluations across logic puzzles, deep search hallucinations, and mode collapse in AI humor generation. | |
| Testing Grok 3 and OpenAI Deep Research on Scaling Laws and FLOPs | 8 | 1 | 1 | 1 | Coogan walks through his extensive prompt test benchmarking FLOP orders of magnitude (E23 to E26) and compares model scaling progression. | |
| Diminishing Marginal Returns in AI Scaling and Report Hallucinations | 7 | 1 | 1 | 2 | Coogan catches conflicting output numbers and hallucinations in ChatGPT Deep Research regarding Grok 3's GPU cluster size while explaining asymptotic benchmark performance. | |
| The Origins of the Musk-Altman Rivalry and Stargate's White House Debut | 5 | 2 | 1 | 1 | The hosts read and analyze the WSJ reporting on Musk's reaction to Altman's Stargate announcement at the White House and their early relationship. | |
| The Breakdown of OpenAI, Stargate's Backstory, and Musk's Hostile Takeover Bid | 6 | 1 | 1 | 1 | Coogan traces the historical breakup of OpenAI's founders, the legal battles, and Musk's ninety-seven billion dollar hostile takeover bid. |
Statements from this episode (22)
Coogan: Reasoning benchmarks are unfair without inference cost and compute time
“There's this big like value trade off between, yeah, if you let the L, if you let the LLM reason for hours and you spend 2000 dollars per query, you can get remarkable results. But then is that really a fair benchmark against something that costs a dollar to i…”
Coogan: xAI Used 20,000 GPUs to Train Grok 3
“XAI famously built a 100,000 GPU cluster in Memphis and now has 200,000 GPUs and used 20,000 of them to train Grok III.”
Coogan: OpenAI is the only AI lab to organically build distribution
“OpenAI is the only AI lab to have organically built its own distribution channel.”
Coogan: xAI raised $6B in November and owns its GPUs
“XAI raised six billion dollars last November and actually owns its GPUs.”
Coogan: Ilya Sutskever's SSI is raising $1B+ at over $30B valuation
“Ilya Sutskiver in Bloomberg is raising more than one billion dollars for his startup at a valuation of over thirty billion.”
Coogan: Green Oaks Is Investing $500M in Safe Superintelligence
“Green Oaks is in. They're putting in 500 mil said the person who asked not to be identified.”
Coogan: SSI Pledges No Product Releases Before Achieving Superintelligence
“So they have pledged not to release a product until it has achieved a safe super intelligence.”
Coogan: Well-funded late entrants like xAI can successfully catch up
“Well, XAI explains why it is absolutely viable to have started late, provided you have the funds catch up quickly.”
Coogan: DeepSeek Proves Chips Aren't the Sole Route to Frontier AI
“On one hand, chips remain the gating factor to the easiest route to a frontier model. On the other hand, this is why deep seek matters. They're not the only route to something that is at least competitive.”
Coogan: The sixth-best AI model can still be a profitable commodity
“Intelligence is, is so valuable and LLMs and AI is so valuable that, that, you know, yeah, even if you're like the sixth best model, you can still sell the intelligence if it's at the frontier level to the market at a market clearing price that still generates…”
Coogan: AI and LLMs will create at least $1 trillion in value
“How, in what world does AI and LLMs not wind up being around a trillion dollars of value? I get that. Like it could go any direction and like how you slice that up. It could be very equal. It could be one winner take all. And you know, Hey, people are gonna ha…”
Coogan: Musk's AI edge is rapid heavy infrastructure execution
“That's really Elon's like edge here for sure.”
Coogan: Grok 3 adapts opinions to confirm users' existing timeline biases
“It's fine tuning on the person who queries it, their timeline. And so Grok three when Elon prompts it knows Elon doesn't like the information. Let's give him an answer that confirms what he believes. And it's the same thing for Paul Calcraft.”
Coogan: Mainstream media will adversarially test LLMs to generate crazy outputs
“You know what's gonna happen here is that the mainstream media is gonna do these tests where they go. You know what they did on like you know, TikTok and Instagram where they'd be like, I followed a bunch of Nazi content, and then it gave me more Nazi content,…”
Coogan: OpenAI's o1 Pro beats R1 and Claude in Catan eval
“The top OpenAI thinking models, O-one Pro get it too, but all of DeepSeek R-one, Gemini, two point O-flash thinking and Claude do not.”
Coogan: 90% of 1,000 ChatGPT jokes were repeats of 25 jokes
“Famously, 90% of the 1000 outputs asking chat GPT for a joke were repetitions of the same 25 jokes.”
Coogan: Frontier AI models face diminishing marginal returns from scaling
“So my basic thesis is that, you know, we're potentially hitting diminishing marginal returns. We're going to see less gains from increase, increased scale.”
Coogan: ChatGPT and Grok 3 disagree on Grok 3's FLOP scale
“ChatGPT claims that Grok three is an E-twenty-six model, but Grok three claims that Grok three is an E-twenty-five model, and so I don't know who's hallucinating here.”
Coogan: Eliminating AI hallucinations may require astronomical compute costs
“I think we will get there. It's just the question of like, you know, what if it's a thousand order of magnitudes?”
Coogan: Sam Altman is likely a billionaire from Stripe alone
“I mean, I think he's a billionaire just from Stripe.”