Jun 12, 2024 · 1h 6m · news
Alex Wang: Why Data Not Compute is the Bottleneck to Foundation Model Performance | E1164 · 20VC with Harry Stebbings
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this deep-dive interview, Scale AI founder Alex Wang outlines why the future of AI model performance depends on highly specialized "frontier data" rather than compute power, while discussing the critical macroeconomic, geopolitical, and organizational shifts shaping the tech industry.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Harry holds 19.8% of the talking time here. How this is scored →
Harry as informed peer 5.4 Guest teaching 3.4 Guest disagreement 1.6 Harry pushing back 3.0
speaking balance: gold is Harry, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 49:15 Alex criticizes traditional media incentives and defense contract coverage
Alex forcefully pushes back on traditional press outlets, accusing them of driving clickbait narratives and unfairly attacking Scale AI for supporting the US military.
Hardest push from Harry ▶ 36:53 Harry forcefully rejects the narrative that China is two years behind in AIHarry bluntly dismisses the claim that China lags two years behind the US in AI capabilities as 'absolute shit', challenging Alex to address China's rapid industrial policy advances.
Biggest teaching moment ▶ 8:05 Alex exposes the pre-training data gap using JP Morgan vs GPT-4 statsAlex educates Harry on the scale of untapped enterprise data by revealing JP Morgan holds 150 petabytes of proprietary data compared to under 1 petabyte used to train GPT-4.
Harry holds his own ▶ 26:55 Harry uses enterprise AI revenue stats and SaaS growth slowdowns to question pricing powerHarry demonstrates superior market knowledge by comparing Accenture's $2.4B GAI revenue with OpenAI's $2B, while citing single-digit growth at Salesforce and MongoDB to challenge software monetization.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Harry as informed peer | Guest teaching | Guest disagreement | Harry pushing back | Why |
|---|---|---|---|---|---|---|
| Is AI Performance Hitting a Compute Wall? | 4 | 3 | 1 | 2 | Harry introduces the compute wall thesis by citing Nvidia's data center revenue surge alongside the lack of a jaw-dropping successor to GPT-4. Alex collaboratively elaborates on the three pillars of AI progress (compute, data, algorithms) to explain the performance plateau. | |
| Defining the Data Wall and the Need for Frontier Data | 5 | 5 | 2 | 2 | Harry demonstrates familiarity with industry concepts, citing Sarah Tavel's framework on software moving from tools to work. Alex educates Harry on data scale by contrasting JP Morgan's 150-petabyte internal dataset with GPT-4's pre-training set of under 1 petabyte. | |
| The AI Reasoning Gap vs. Human Intelligence | 5 | 4 | 1 | 2 | Harry references a recent conversation with a prominent CTO regarding solving AI reasoning and asks about synthetic data and AI trainer job roles. Alex explains the difference between human general intelligence and machine pattern matching using an autonomous vehicle safety driver analogy. | |
| Enterprise Data Mining and Passive Data Capture | 6 | 3 | 1 | 3 | Harry shows technical domain knowledge by bringing up Dan Siroker's Limitless wearable hardware and enterprise process mining via RPA/UiPath workflows. Alex outlines the distinction between one-time enterprise data mining and ongoing forward data production. | |
| Data as the Ultimate Moat for Foundation Models | 6 | 2 | 1 | 2 | Harry brings up specific content licensing deals between OpenAI, the Financial Times, and Axel Springer to probe data exclusivity. Alex agrees, noting data is the only durable moat among the three pillars compared to leakable algorithms or buyable compute. | |
| The Rise of On-Premise AI and Data Privacy | 7 | 3 | 2 | 4 | Harry demonstrates strong market expertise by contrasting Accenture's $2.4B Generative AI revenue against OpenAI's $2B, while citing single-digit growth at Salesforce and MongoDB to challenge AI monetization in SaaS. Alex references Andy Grove's High Output Management to analyze value capture across the AI stack. | |
| The Death of Per-Seat Pricing in the Agentic Era | 5 | 4 | 1 | 3 | Harry presses on whether European and UK consumer data protections risk stifling AI innovation relative to global competitors. Alex lays out specific policy ideas for pro-data regulation, such as pooling anonymized aerospace safety data and updating healthcare HIPAA provisions. | |
| The China AI Threat and Geopolitical National Security | 6 | 5 | 3 | 6 | Harry forcefully rejects claims that China is two years behind the US in AI as 'absolute shit', bringing up Chinese industrial velocity and EV market data. Alex validates Harry's point, citing 01.AI's Yi-Large model benchmarking near GPT-4o, and warns that AI could surpass nuclear weapons as a military asset. | |
| The Multi-Billion Dollar Future of Foundation Models | 6 | 3 | 3 | 4 | Harry interrogates media incentives and founder branding strategies, asking Alex about his statement that 'the best PR is no PR'. Alex strongly criticizes traditional media for sensationalism and unfair coverage of Scale AI's defense contracts with the US Department of Defense. | |
| Hiring for Intensity: Selecting Talent That Truly Cares | 5 | 3 | 2 | 3 | Harry presses Alex on operational hiring details, asking how an 800-person company maintains an elite bar and what percentage of hire recommendations Alex overrides. Alex reveals his 'Navy SEALs vs Navy' hiring methodology and notes he personally overrides 25-30% of hiring manager decisions. | |
| Alex Wang's Biggest Leadership Mistake: The Trap of Team Hypergrowth | 5 | 2 | 0 | 2 | Harry shares a personal leadership vulnerability regarding managing out of fear versus freedom to prompt Alex. Alex admits his biggest mistake was assuming company hypergrowth required team hypergrowth, which diluted talent density as Scale grew from 150 to over 700 employees. | |
| The Paradox of Brand Heat and Talent Ecosystems | 5 | 4 | 2 | 3 | Harry brings up cycles of brand heat in tech companies and points to OpenAI's London office expansion. Alex relays private insights from Stripe co-founder Patrick Collison and Airbnb CEO Brian Chesky about avoiding clout-seeking hires brought in during peak brand heat. | |
| Quick-Fire Round: AGI Misconceptions, Leadership, and the AI Hype Cycle | 5 | 3 | 2 | 3 | In a rapid-fire sequence, Harry asks about AGI misconceptions, dream board members, US election predictions, and IPO plans. Alex draws a direct comparison between today's generative AI hype promises and the earlier hype-and-trough cycle of autonomous vehicles. |
Statements from this episode (31)
Alex Wang: AI could be a greater military asset than nuclear weapons
“At its core, this AI technology has the potential to be one of the greatest military assets that humanity has ever seen. Potentially even more of a military asset than nukes.”
Alex Wang: China or Russia would use AGI to conquer if developed first
“Let's say China or Russia had AGI today and the United States didn't, I would imagine they would use that to conquer.”
Alex Wang: China's centralized industrial policy gives it an AI advantage
“The CCP's system is incredibly good at taking very aggressive centralized action and centralized industrial policy to drive forward critical industries. They have a clear shot at racing forward.”
Alex Wang: AI performance plateau is caused by hitting a data wall
“And I think a lot of the plateau that we've recently seen can almost be explained at a very high level from hitting kind of a data wall.”
Alex Wang: GPT-4 was trained on nearly all internet data
“GPT-IV was a model basically trained on nearly all of the internet and using a huge amount of computational capabilities.”
Wang: AI model training has exhausted all available internet data
“At a super super high level I think we've used up all the easy data. We've used up all of the internet data.”
Wang: JP Morgan's internal data is 150 petabytes versus GPT-4's sub-petabyte dataset
“JP Morgan's proprietary internal data set is a 150 petabytes. The GPT-IV was trained on an internet data set that was less than one petabyte.”
Alex Wang: Current AI Systems Lack Zero-Shot General Intelligence
“No AI system today would be able to do that level of sort of, ah, you know, drag and drop in one situation to another situation and figure out what's going on.”
Wang: Overwhelming Models With Scenario Data Solves AI Reasoning
“It's like you need data for every scenario where you want these models to reason well in, You just need to overwhelm them with data in all those scenarios, and you're gonna get models that can reason really well.”
Alex Wang: Contributing AI data is among highest-leverage human jobs
“I think this process of contributing data to AI is actually one of the highest leverage jobs that humans can have.”
Wang: Only the most sophisticated companies will mine internal data within five years
“I don't think everyone will, but certainly the most sophisticated companies will.”
Wang: Process mining and passive data won't advance AI model frontiers
“Both, you know, enterprise process mining and you know, for lack of a better term, sort of like consumer data collection, those are all gonna produce valuable data sets, but they're not gonna produce the data that's actually gonna push the models forward.”
Wang: AI competition will shift from GPU access to proprietary data rights
“Right now if you know, in San Francisco, the big, you know, researchers and the big CEOs brag about how many GPUs they have. You know, they're the sort of the biggest indicator of how serious they are about AI is, is how many GPUs they have. But I think in the…”
Wang: Most serious enterprises will adopt on-premise AI models
“And this is actually why I think there's a very there's a very big sort of opportunity for whether it's open source models or the llama models or the mistral models or whatnot, basically these models that can go on prem and that enterprises can take. And then …”
Stebbings: AI services will generate more revenue than AI models
“I think AI services will actually create more revenue over the next five years than AI models.”
Alex Wang: AI value will accrue above and below foundation models
“I think that the models themselves there's so much competition there that I don't know if, I don't know how much value accrues at literally the model itself, but the everything above the model and everything below the model, I feel very confident there will be…”
Alex Wang: Falling software costs will shift enterprises to custom applications
“Especially that the software production costs and software creation costs are going down so dramatically, we're going to end up Moving towards a world where more and more of the software that enterprises consume are going to be customized and custom built and …”
Alex Wang: Enterprise software will shift from per-seat to consumption pricing
“The reason that per seat pricing doesn't make sense going into the future is that at an enterprise today, certainly most of the productive work is done by their employees, done by people. But in a future where you imagine more and more of the work is done by A…”
Wang: Permissive data access regulations are compatible with liberal democracy
“My personal belief, I don't think that more permissive regulations around data are at odds with being a liberal democracy. More sort of liberal data, data access provisions Are in fact very, very compatible with being a liberal democracy.”
Wang: HIPAA rules currently block patient data from training AI models
“Right now you know, HIPAA and PII and all the PII regulations will more or less prevent patient data from being used to train you know, AI models.”
Alex Wang: Chinese LLM capabilities are nearly neck and neck with US
“Chinese LLM and AI capabilities are, I would say right now, pretty close to neck and neck with us capabilities.”
Alex Wang: China executes industrial scaling better than any other economy
“The CCP approach to industrial policy is not the most innovative, but once an industry has been established and it's about, you know, turning the crank, they are better at turning the crank than any other economy in the world.”
Alex Wang: Most advanced AI models must remain closed for national security
“I think we need to think about the most cutting edge and the most advanced systems. Those we will want to ensure are closed for geopolitical reasons, for military reasons, for whatever reasons, like as we develop systems that are genuinely so, so powerful, we …”
Alex Wang: AI foundation models will cost $100B+ to train within 10 years
“I think in 10 years time, maybe they'll cost tens or hundreds of billions. And so there's just not very many entities that have that much discretion and capital to invest into these AI models. So naturally what will happen over time is AI effort, the foundatio…”
Alex Wang: Founders should ignore traditional press and use direct distribution
“I would argue we're in an era now where they shouldn't and they should think about they should think about what is a, what is an interesting point of view they can have, and what are the, what is the most pure way to get that point of view across?”
Alex Wang: Startups win because small teams care 10x more than corporate employees
“That's how startups work fundamentally, is that you have these small teams of people who each care 10 times more than the average employee, 10 or a hundred times more than the average employee inside a big company, and so they end up, you know, you end up just…”
Alex Wang personally approves every single hire at Scale AI
“At this point in the company, still, I approve every hire, so I will look at The, I will either indirectly interview or look at the interview feedback and look at the understand every single person who we hire to ensure that we're keeping an exceptionally high…”
Alex Wang rejects team hiring recommendations up to 30% of the time
“I would say maybe on average, 25 to 30%, like a lot.”
Alex Wang: Hypergrowth hiring makes it impossible to maintain a high talent bar
“When you hire that quickly, it is impossible to do what we've just been talking about, which is maintaining this high bar and maintaining this sort of feeling of excellence within the team.”
Wang: Generative AI risks an industry hangover due to overpromising
“I think this is one of the big concerns I have about generative AI, which is I hope not, but the same thing might happen again, which is that we have these really big promises that are starting to, that get made about the technology that get divorced from tech…”
Shorts cut from this episode
▶ Three Competitive Advantages Of AI 🤖 · 20VC with Harry Steb (@19:32)
▶ Attracting top talent: Advice from Stripe CEO Patrick Collis (@57:41)
▶ This billionaire CEO reviews EVERY hire in his company...🤯 (@53:15)
▶ AI more powerful than nuclear weapons? 😳 · 20VC with Harry (@0:00)
▶ Biggest mistake in AI today 🤖❌ · 20VC with Harry Stebbings (@1:02:58)
▶ Is China winning AI? 🇨🇳 · 20VC with Harry Stebbings (@37:31)