Sep 15, 2023 · 31m · 20vc
20VC: The Biggest AI Leaders on What Matters More; Model Size or Data Size & Where Does The Value in AI Accrue; to Startups or to Incumbents
Harry Stebbings · 10m spoken Richard Socher · 7m spoken Douwe Kiela · 3m spoken Alex Lebrun · 2m spoken Emad Mostaque · 1m spoken Cris Valenzuela · 1m spoken Noam Shazeer · 1m spoken Tomasz Tunguz · 1m spoken Sarah Guo · 39s spoken Clément Delangue · 36s spoken
0:00 / 0:00
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
This compilation episode of The TwentyVC Podcast brings together top artificial intelligence founders and investors to analyze whether compute scaling or data volume drives AI performance, and whether long-term value will accrue to fast-moving startups or enterprise incumbents.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Harry holds 35.3% of the talking time here. How this is scored →
Harry as informed peer 4.9 Guest teaching 4.6 Guest disagreement 2.9 Harry pushing back 3.6
speaking balance: gold is Harry, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 23:39 Alex LeBrun dismisses incumbent AI features as superficial magic dust
Alex forcefully rejects the idea that incumbent feature additions represent true disruption, arguing that companies like Notion and Google Docs will eventually be destroyed by entirely new paradigms.
Hardest push from Harry ▶ 23:25 Harry refuses the premise that incumbents are moving slowlyHarry directly interrupts and challenges Alex LeBrun's assertion that incumbents are universally slow by presenting specific, fast-moving companies like Notion, Adobe, and Navan.
Biggest teaching moment ▶ 6:39 Richard Socher explains parameter scale and world knowledge encodingRichard uses a vivid linear regression comparison and next-token prediction example across geography to educate the host on why billions of parameters are fundamentally required to capture nuanced knowledge.
Harry holds his own ▶ 27:17 Harry cites Tom Tunguz data to challenge Emad's thin layer claimHarry counters Emad Mostaque's dismissal of application-layer value by citing specific data from Tom Tunguz showing that application layers capture trillions of dollars across diverse companies.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Harry as informed peer | Guest teaching | Guest disagreement | Harry pushing back | Why |
|---|---|---|---|---|---|---|
| Noam Shazeer on Compute and Model Scale | 4 | 3 | 2 | 3 | Harry Stebbings prompts Noam Shazeer and Cris Valenzuela on compute constraints and model verticalization versus horizontal consolidation. Cris gently rejects the premise of a single model ruling the ecosystem, comparing AI tools to e-commerce diversity. | |
| Richard Socher on Parameters, Data Distribution, and Retrieval Backends | 5 | 7 | 2 | 4 | Harry raises venture community skepticism about AI startups being thin wrappers around base models. Richard Socher provides an extensive technical masterclass on parameter scaling, world-knowledge encoding, and search retrieval backends to dismantle the thin-wrapper dismissal. | |
| Douwe Kiela on Data Efficiency and Optimal Compute Allocation | 4 | 5 | 3 | 2 | Douwe Kiela clarifies compute optimality and sample efficiency trade-offs when training smaller models on larger data. Emad Mostaque adds provocative commentary on national datasets and algorithmic bias. | |
| Sarah Guo on Startup Execution Speed vs. Incumbent Data Moats | 5 | 5 | 2 | 3 | Sarah Guo, Clem Delangue, and Douwe Kiela explore startup velocity versus incumbent distribution. Harry presses Douwe on why venture capitalists reject startups lacking proprietary data moats, and Douwe explains LLM data efficiency. | |
| Richard Socher on Distribution Battles and the Innovator's Dilemma | 4 | 4 | 2 | 2 | Harry asks Richard Socher about the race between startup distribution and incumbent innovation. Richard articulates Google's Innovator's Dilemma around protecting a five-hundred-million-dollar-a-day advertising business. | |
| Alex LeBrun on Feature Layering vs. New Product Paradigms | 6 | 4 | 5 | 6 | When Alex LeBrun argues incumbents are too slow, Harry immediately pushes back with concrete counterexamples including Adobe, Notion, and Navan. Alex concedes the point but dismisses incumbent implementations as merely sprinkling magic AI dust onto legacy products. | |
| Tom Tunguz on Execution Moats and Startup Agility | 6 | 4 | 4 | 5 | Tomasz Tunguz discusses execution moats before Emad Mostaque claims foundation models will heavily consolidate. Harry counters Emad's skepticism about application layers by citing Tom Tunguz's quantitative market-cap distribution data. |
Statements from this episode (25)
Noam Shazeer: Compute and Model Size Constrain AI Progress More Than Data
“Yeah, probably the size of the model is the bigger challenge. We can get a lot of data, but actually the number one thing that's important is how much computation you do to train it.”
Character.ai Spent $2 Million in Compute to Train Its Serving Model
“The model we're serving now, we trained last summer and spent about two million dollars worth of compute cycles doing it.”
Chris Valenzuela: No Single Monolithic AI Model Will Dominate the Market
“I don't think there's going to be a single model to rule them all.”
Chris Valenzuela: AI Models Are Not Moats; Talent and Speed Win
“I think models are not a mode. Models eventually don't matter. What matters most is the people building those models. And how fast can you change and learn from those models?”
Richard Socher: Enterprise Incumbents Hold a Key Advantage in Proprietary Data
“So it's complicated in the sense that unsupervised data, just raw internet text is easily accessible, but there's still a lot of data sets out there that are not out there. They're actually stored in a private databases. And indeed, if you want to answer custo…”
Richard Socher: AI Startups Can Build Moats Through Distribution and Partnerships
“Turns out you can have moat other than your backend AI model, right? It's distribution, it's partnerships. Your sales funnel processes and so on.”
Douwe Kiela: AI Model Parameter Sizes Will Stop Growing Rapidly
“I think Sam Altman had this interesting quote where he was saying that he thought models would stop growing in size. GPT-IV kind of hit this ceiling. I think that's probably right, but not really because size doesn't matter. It's just that data size matters ev…”
Douwe Kiela: Training Smaller AI Models on More Data Increases Efficiency
“If you train a smaller model on more data for longer, then you get a better model. So you get more bang for your buck if you train it on more data rather than having more parameters.”
Emad Mostaque: Why I Signed the Open Letter for an AI Pause
“The reason I signed that letter, because I think there's a six month pause to get all of our shit together before things go completely insane, and next year this is everywhere, and everyone's investing in everything, and it's just absolute chaos.”
Emad Mostaque: Every Nation Will Need Sovereign Datasets and Open Models
“As part of that, every nation will need their own data sets, which again, have from broadcaster data. They will need their own open models that can stimulate innovation internally as well.”
Emad Mostaque: No Current AI Models Will Be Used in a Year
“The reality is no models that are out today will be used in a year.”
Sarah Guo: Speed Is the Only Real Advantage Startups Have
“Classically, the only real advantage startups have is speed, and speed actually might matter more than ever when the environment seems to be moving at warp speed”
Sarah Guo: Incumbent Data Moats in AI Are Overhyped
“On the incumbent advantage side, much ado has been made about this idea of a data moat. But honestly, there's a lot of data out there and entrepreneurs are incredibly creative about collecting it and increasingly about generating it.”
Clément Delangue: Deep AI Startups Can Execute 100x Better Than Incumbents
“I think if you were thinking about AI as AI APIs, I agree. If you're thinking of AI as a more radical paradigm switch to build technology, right? And if you think about an AI startup as a company that is actually training models, creating new architectures, op…”
Douwe Kiela: Meta's Original LLaMA Was Trained Entirely on Open Data
“So the LAMA model was not trained on any proprietary data. It was just trained on open data on the web.”
Douwe Kiela: Deep Tech AI Startups Must Build Data Flywheels
“If you want to build a deep tech AI startup, then you really want to get a big data flywheel going. You want to start with a lot of data and then have a way to generate lots more data and that data is going to be your moat.”
Douwe Kiela: GPT-4 Will Disrupt Data Annotators Like Mechanical Turk
“One of the use cases I've been seeing now for GPT-IV is actually that people are using it to generate data and then they're training on that data with cheaper models. So GPT-IV might end up disrupting, not knowledge workers necessarily, but it might just disru…”
Richard Socher: Bing and Google Copied YouChat Months After Its Launch
“We've also seen Bing and Google copy what we have launched late last year with you chat. They've copied us in the sense that we've launched it earlier and then they launched something very similar three to four months after.”
Richard Socher: Google's Ad Revenue Creates an Innovator's Dilemma Against AI
“That kind of big change will be hard for Google too, because they make five hundred million dollars a day with privacy invading advertisements on that page. And so you don't just willy nilly change most of that page and you get rid of the five, six ads that ar…”
Alex LeBrun: AI Paradigms Will Eventually Displace Notion and Google Docs
“Probably something will Come and destroy Notion and Google Docs and all of them with a totally new paradigm that is made possible by AI.”
Alex LeBrun: LLM Capabilities Could Not Be Predicted Before Scaling
“The reason is nobody could predict that LLMs would be so useful and powerful before you try to run at this scale.”
Emad Mostaque: Only Five Foundation Model Companies Will Survive by 2028
“I think that there's only going to be five or six foundation model companies in the world in three years, five years.”