Patel: Frontier AI cluster costs have scaled from $100M to $10B
Dylan Patel · Generative AI 101: Tokens, Pre-training, Fine-tuning, Reasoning — With SemiAnalysis CEO Dylan Patel · Apr 23, 2025 · at 31:54
Dylan Patel, CEO of SemiAnalysis, details the physical and capital scaling trajectory of AI training supercomputers from GPT-4 to OpenAI's planned Stargate cluster in Texas.
0:00 / 0:32exact quote · 32.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“For GPT-IV, it was a few hundred million dollars and it's one building full of GPUs, too. GPT-IV 4.5 and the reasoning models, like, oh, one, oh, three were done in a, in three buildings on the same site, and, you know, billions of dollars to, hey, these next generation things that people are making are tens of billions of dollars like OpenAI's data center in Texas called Stargate, right? With Crusoe and Oracle and et cetera, right?”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dylan Patel
Patel: Most AI models lean left due to Bay Area origins
“Most AI models are made in the Bay Area, so they tend to just be left leaning, right? But also the internet in general is a little bit left leaning because it skews younger than older.”
Patel: Building cheaper AI requires massive frontier models for synthetic data
“You can't actually make that cheaper model without making the better model, bigger model. So you can generate data to help you make the cheaper model, right?”
Patel: $10B AI data centers aim to automate software engineering, not chatbots
“No one is trying to make with these, you know, with these ten billion dollar data centers, they're not trying to make chat models, right? They're not trying to make models that people chat with, just to be clear, right? They're trying to solve things like soft…”
Patel: OpenAI's Orion training run failed to reach GPT-5 performance levels
“There were hopes that Orion could be used for GPT-V but its improvement was, like, not enough to be, like, really a GPT-V. Furthermore, it was trained on the classical method, which is, like which is a ton of pre-training, and then some reinforcement learning …”
Patel: Language is a representation for reasoning, not human thought itself
“Language is not actually how our brain thinks. It's just a representation for which it to, you know, reason over.”
Patel: AI models ingest dangerous data during pre-training for world knowledge
“So you don't want to just filter out everything so that the model doesn't know anything about it but at the same time, you don't want it to output, you know, how to build a bomb so there's like a fine balance here, and that's why pre-training is defined as pre…”