Aug 19, 2024 · 1h 3m · news
Aidan Gomez: What No One Understands About Foundation Models | E1191 · 20VC with Harry Stebbings
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode, Cohere co-founder and CEO Aidan Gomez discusses the shifting paradigms of generative AI, outlining why brute-force compute scaling must give way to clever data innovations, secure enterprise integrations, and specialized architectures. He shares business insights on navigating industry consolidation, competing with cloud hyperscalers, and leveraging virtual private deployments to unlock the next wave of global labor productivity.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Harry holds 21.8% of the talking time here. How this is scored →
Harry as informed peer 4.2 Guest teaching 4.8 Guest disagreement 2.5 Harry pushing back 3.1
speaking balance: gold is Harry, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 50:29 Gomez Rejects Host's AI Parenting Fears
Gomez forcefully rejects Harry's anxiety about children interacting with AI agents, starting with 'You might actually be wrong' and arguing AI companions provide safe, empathetic learning without replacing human partners.
Hardest push from Harry ▶ 27:55 Host Challenges Model Cost Escalation ModelHarry directly challenges Gomez's claim about model economics by pointing out that traditional software updates cost incremental millions while foundation model generations require exponential order-of-magnitude jumps from $3B to $5B.
Biggest teaching moment ▶ 26:09 Gomez Clarifies 'FLOPs' TerminologyAfter Harry humorously confuses 'FLOPs' with British slang for a blunder, Gomez educates the host on floating point operations as fundamental compute units tied to parameter counts.
Harry holds his own ▶ 35:40 Host Cites Expert Quote on OpenAI's AGI StrategyHarry demonstrates deep preparation by citing Ethan Mollick's podcast claim that OpenAI neglects useful products like Code Interpreter due to a singular focus on AGI, prompting a nuanced rebuttal from Gomez.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Harry as informed peer | Guest teaching | Guest disagreement | Harry pushing back | Why |
|---|---|---|---|---|---|---|
| Gomez's Childhood in Rural Ontario and Path to CS | 2 | 3 | 1 | 1 | Harry opens with warm rapport-building questions about Gomez's upbringing in rural Ontario and draws a connection between childhood video gaming and successful founders. Gomez responds collaboratively, noting how gaming fosters resilience and curriculum learning analogies, though clarifying that curriculum learning actually failed in machine learning. | |
| The Limits of Scaling and the Rise of Efficient Models | 3 | 5 | 2 | 2 | Harry asks a direct question about whether compute scaling remains the sole rate limit for model quality. Gomez educates the host by framing pure compute scaling as the dumbest yet most reliable strategy, noting how parameter efficiency has dramatically outpaced raw compute scaling. | |
| Data Innovations, Reasoning, and Synthetic Data | 4 | 6 | 2 | 3 | Harry cites OpenAI's $3 billion annual burn to press Gomez on how non-hyperscalers can survive, asking for explicit definitions of data vs method innovations. Gomez details synthetic data generation, web parsing, and reasoning datasets, while explaining enterprise privacy barriers. | |
| The Race to Zero: Business Models and Monetization in AI | 4 | 5 | 2 | 3 | Harry brings up market dynamics like Meta releasing models for free and price dumping by OpenAI, questioning if API providers are in a race to zero. Gomez agrees that pure API model selling will suffer zero margins and educates on how value accrues at the hardware and application layers. | |
| Hardware Strategy: Chip Spend, Cloud Partners, and Infrastructure | 5 | 4 | 2 | 3 | Harry asks informed questions regarding Cohere's chip expenditures, multi-cloud strategy, GPU stockpiling, and potential data center buildouts. Gomez outlines the stark structural difference between heterogeneous inference options and monopolized training compute platforms. | |
| The Transformer Paper and the Turning Point of ChatGPT | 4 | 3 | 1 | 2 | Harry demonstrates strong prep by bringing up Gomez's co-authorship of the 2017 Transformer paper and asks about turning points. Gomez shares historical context on early language modeling at Google Brain and discusses why text chat and voice are superior user interfaces over GUIs. | |
| The Cost of Human Expertise and Demystifying 'FLOPs' | 2 | 7 | 2 | 2 | Harry asks about talent bottlenecks and confuses technical compute terminology, mistaking 'FLOPs' for British slang for a blunder. Gomez educates the host on domain expert scarcity for model tuning and explains floating point operations. | |
| Why There Is No Market for Last Year's Model | 5 | 5 | 4 | 5 | When Harry suggests lower compute costs enable new startups to build models, Gomez firmly counters that there is no market for last year's model. Harry pushes back on Gomez's framing by comparing traditional software upgrade costs to order-of-magnitude foundation model cost jumps. | |
| The Cloud Land Grab, Consolidation, and Cohere's Valuation | 6 | 4 | 3 | 5 | Harry drills into market consolidation, citing recent aqua-hires of Adept and Inflection by tech giants, and asks if Cohere's $5.5B valuation creates intense revenue multiple pressure. Gomez defends Cohere's position by highlighting the danger of becoming a cloud provider subsidiary. | |
| Admiring OpenAI: Ilya Sutskever's Conviction and the Product Pivot | 6 | 5 | 3 | 4 | Harry quotes professor Ethan Mollick's claim that OpenAI neglects useful products in favor of singular AGI pursuit. Gomez pushes back on this premise, explaining that OpenAI has fundamentally shifted into a consumer product company. | |
| AI Commercialization, Margin Compression, and Canva's Strategy | 6 | 4 | 1 | 3 | Harry cites a case study from Canva's founders detailing margin compression from unmonetized AI features to ask about enterprise adoption barriers. Gomez categorizes market pricing tactics and outlines Cohere's private VPC deployment model. | |
| AI Hallucinations and Retrieval-Augmented Generation (RAG) | 2 | 6 | 2 | 2 | Harry asks what enterprises misunderstand about AI hallucinations. Gomez educates the host by comparing model hallucinations to human errors and explaining how Retrieval-Augmented Generation (RAG) resolves accuracy and attribution problems. | |
| Mainstream Enterprise Budgets and Workforce Augmentation | 4 | 5 | 2 | 3 | Harry asks whether enterprise budgets are transitioning from proof-of-concepts into production and brings up Microsoft Copilot. Gomez explains that Copilot is structurally limited by being siloed within the Microsoft ecosystem, whereas enterprises require agnostic orchestration across tools like Salesforce and SAP. | |
| The Promise of AI Agents and the Edge of Model Builders | 5 | 5 | 3 | 4 | Harry asks about agent hype, prompting Gomez to argue that agentic software will be dominated by model builders rather than third-party wrappers. Harry pushes back on this skepticism toward application platforms by defending Marc Benioff and Salesforce's enterprise stickiness. | |
| The Battle for AI Talent and OpenAI's Scaling Hypothesis | 4 | 5 | 3 | 2 | Harry asks about research talent concentration and market narratives surrounding model progress plateauing. Gomez rejects the plateau thesis, explaining that upcoming gains will stem from reasoning, search, and planning paradigms rather than pure parameter scaling. | |
| Human-AI Relationships, Labor Augmentation, and Content Moderation | 4 | 6 | 6 | 5 | Harry voices deep concern about children forming emotional attachments to AI agents instead of humans. Gomez directly challenges Harry ('You might actually be wrong') and argues AI companions will be safe, patient learning partners while human relationships remain irreplaceable. Harry pushes back on workforce displacement by pointing to Klarna's customer service cuts. | |
| General-Purpose Robotics and Foundation Model Planners | 5 | 5 | 3 | 3 | Harry initiates a rapid-fire round covering Gomez's technical mindset shifts, total fundraising ($1B), broken personal economic perspectives, and Jeff Hinton vs Yann LeCun. Gomez explains model sensitivity to data quality and details foundation models as dynamic planners for general-purpose robotics. | |
| European Tech Culture, In-Person Work, and the Abundance Goal | 5 | 4 | 3 | 3 | Harry quotes Delian Asparouhov's claim that Western Europe faces decline, asking Gomez about building engineering teams in London. Gomez agrees continental Europe's culture is hostile to tech through heavy regulation, while praising UK tech optimism and advocating for productivity growth. |
Statements from this episode (52)
Aidan Gomez: There is no market for last year's AI models
“The reality of the matter is there's no market for last year's model.”
Aidan Gomez: AI ecosystem will feature both vertical and horizontal models
“I think we'll continuously exist in a world of multiple models, some focused and verticalized, others completely horizontal.”
Aidan Gomez: AI market will definitely undergo consolidation
“There's going to be a consolidation in the space for sure.”
Aidan Gomez: Becoming dependent on cloud providers is dangerous for AI startups
“It's really dangerous when you make yourself a subsidiary of your cloud provider.”
Aidan Gomez: Video games teach founders resilience and willingness to grind
“Video games teach something to you. You're much more Willing to grind. Right. To just do repetitive, difficult, painful things towards some broader goal. So that sort of resilience I think is important. And then also the fact that you can respond. Like you get…”
Aidan Gomez: Curriculum learning has failed in machine learning
“What's funny is that curriculum learning has actually failed in machine learning. We don't really do curriculum learning. It's just throw the hardest material and the easiest material all at the same time and let the model figure it out.”
Gomez: 13-billion-parameter models now outperform original 1.7-trillion GPT-4
“GPT-IV, if it's true what they say, and it's 1.7 trillion parameters, this big MOE, we have models that are better than that model that are like, thirteen billion parameters.”
Gomez: AI requires exponential compute scaling for linear intelligence gains long-term
“Yeah, I mean, I think it certainly requires exponential input. You know, you need to continuously be doubling your compute in order to sustain linear gains in, in intelligence. But I think that probably goes on for a very, very, very long time.”
Gomez: Pure compute scaling requires becoming a tech giant subsidiary
“If you're just doing the scaling project, you have to be one of those, or you have to be an effective subsidiary of one of those companies.”
Gomez: Data improvements drive nearly all open-source AI model gains
“Pretty much all of the major gains that we've seen in the open source space have come from data improvements.”
Gomez: Cohere does not train AI models on customer enterprise data
“Especially with enterprises, they never let you train on their data. And so we can't train on any of our customers' data.”
Gomez: Current LLM API market is dominated by synthetic data generation
“The current LLM API market is dominated by synthetic data.”
Gomez: Cohere Will Soon Expand Its Product Suite Beyond Model APIs
“I don't want to name names, but let's say Coherit right now only sells models. We have an API, and you can access our models through that API. I think that that will change soon. There are going to be changes in the product landscape and what we offer to sort …”
Gomez: Selling Standalone AI Models Will Become a Zero-Margin Business
“If you're only selling models, it's going to be difficult because it's going to be like a zero margin business because there's so much price dumping.”
Gomez: AI Value Currently Accrues at the Chip and Application Layers
“Value is accruing beneath, like at the chip layer, because everyone is spending insane amounts of money on chips to build these models in the first place, and then above, at the application layer, where you see stuff like let's say ChatGPT, which is charged on…”
Gomez: Choice in the AI chip market will expand faster than expected
“Right now, chips are just exceptionally high margin, and there's very, very little choice in the market. That's changing. I think it's gonna change faster than other people think. But yeah, I, I'm very confident.”
Gomez: Google TPUs are now viable for super large-scale AI training
“You can definitely train big models on TPUs. Those are actually now a usable platform for super large scale model training, and I think Google has proven that quite convincingly.”
Gomez: Cohere won't build data centers because cloud pricing is cheaper
“We're an economically rational actor, and so if it's cheaper for us to build out our own data centers, we'll go do that. We've run the numbers, and we feel confident that the price we're getting from our providers makes that not a really attractive path.”
Gomez: Industry took two to three years to realize language model scaling worked
“With language modeling and the whole scaling project, I thought the world would catch on way faster to that piece. It started to become really obvious, but then it was two, three years before everyone woke up, and it sort of hit the world.”
Gomez: GUIs Are Not Dead and Should Not Be Fully Replaced by Chat
“Chat as an interface onto everything, I don't think makes sense. I don't want to have to type out explicitly my instructions to get stuff done. Like sometimes I just want to click some buttons and go through a GUI and get the job done. So yeah, I don't think l…”
Gomez: 2017 Language Models Generated Wikipedia Pages as Convincing as Humans
“That happened, like, in 2017, shortly after we submitted the paper. We started training language models on Wikipedia and we sampled from those models, and it could write Wikipedia pages as convincing as a human page.”
Gomez: Industry Investment in Voice AI Interfaces Is Entirely Justified
“Absolutely. Like anyone who has tried having a voice-based conversation with one of these models, it's like a stunning experience. Like you're kind of left In shock when you hear the model exhibiting emotion and inflection and, you know, you hear it breathe to…”
Gomez: Building last year's AI model gets 10x to 100x cheaper annually
“It becomes cheaper to build last year's model by like a factor of 10 or a hundred each year. We just get better data, cheaper compute. So yeah, it definitely lowers the barrier to the previous generation of models.”
Gomez: Non-experts cannot perceive improvements between frontier AI model generations
“Because these models are getting smarter, humanity's ability to distinguish between them, or not humanity, but each individual's ability to distinguish between them becomes way harder. You can't tell the difference between generations, because you're not enoug…”
Gomez: Cohere avoids the hyper-inflated valuations of AI comparables
“We are actually in a dramatically better position than a lot of our comparables. Because our valuation is not at the crazy state that a lot of others are.”
Gomez: Ilya Sutskever envisioned compute scaling years before GPT-1
“They pave the way. Like, just sort of like a irrational conviction to this vision of scaling, and I think that, that was driven by, I remember talking to Ilya about this stuff way before GPT-One, you know, like in the Transformer times, around that time becaus…”
Gomez: OpenAI's AGI effort is taking a backseat to consumer products
“At least like lately, or in the new OpenAI they're like a product company. They're like hardcore building a consumer product. That is their objective, and it's working. People love that product. It's, you know, a household name at this point. So I think in the…”
Gomez: Software companies shouldn't fear AI margin compression as costs plunge
“For folks like Canva who are keeping the same price I mean, I think it's a good bet. They want to grow their user base. They want to expand their user set. Just give them the most useful product possible. At the moment, don't worry about margins because the co…”
Gomez: Trust and security are the primary blockers for enterprise AI
“It's mostly trust. In the technology. So security. Everyone is very sketched out by the current state of things.”
Gomez: Financial services firms are pulling back from cloud to build data centers
“When I speak to folks, It's super conflicted in financial services. Yeah. People are pulling away from cloud. They're pulling away from cloud. They're building out their own data center capacity. Everywhere else still seems to be, we need to migrate to cloud.”
Gomez: AI hallucination rates are falling fast, but will never reach zero
“The models definitely do hallucinate. The hallucination rates have been dropping dramatically, but they'll, they'll always have some chance of making stuff up or getting something wrong.”
Gomez: Enterprise concern over AI hallucinations is overblown relative to human error
“We exist in a world with humans, and humans hallucinate. Constantly. We get stuff wrong, we, you know, misremember things, and so we exist in a world that's robust to error, and so I think there's too much emphasis on that.”
Gomez: Enterprise AI spending has shifted from POCs to production deployment
“So last year, a hundred percent. It was like the year of the proof of concept. Everyone was sort of testing it out, playing around with it. But recently there's been a big shift to Urgency to get this tech into production. I think a lot of enterprises are scar…”
Gomez: Employee augmentation is the top enterprise AI use case
“I think it's employee augmentation. It's these models becoming like a partner or a colleague to your entire workforce. That's the most popular use case.”
Gomez: Microsoft Copilot will not achieve complete enterprise workforce augmentation
“I think Copilot is great, and it's like the right idea of augmenting a workforce with a, with an assistant but it's siloed, again, within an ecosystem, so it's, it plugs into Office and, you know, the Microsoft suite of products, but enterprises don't just use…”
Aidan Gomez: AI agent developers are disadvantaged without owning model architecture
“If you're not able to actually transform the model to be better at the thing that you care about, if you're not the one building the model, if you're just a consumer of the model you're structurally disadvantaged to build that product.”
Aidan Gomez: Enterprise software is extremely sticky and persists for decades
“The other thing is that you forget how sticky enterprise software is. There's not a lot of like mass displacement of enterprise software. It kind of just stays for decades. It's really hard to displace an enterprise software company.”
Gomez: Cohere Has Top AI Researchers, Ecosystem Talent Is Now Distributed
“Cohere. But other than Cohere, I think it's quite distributed at this stage. It used to be very concentrated. It used to be like Google Brain.”
Gomez: Google Brain Trained Language Models Two Weeks After Transformer Paper
“Well, they weren't in the sense that, like, two weeks after we released the Transformer paper, we started training language models. So we, like, technologically and research-wise Google brain was certainly not behind.”
Gomez: OpenAI's Best Move Was Pursuing the Scaling Hypothesis
“The scale hypothesis, for sure. Like, just that scaling is gonna sustain, and that we should continue to 10 x, 10 x, 10 x, 10 x. Yeah, so many people didn't believe in that. There was so much pushback on it. It was just like, Such a stupid, superfluous effort …”
Gomez: Believing AI progress has plateaued is completely wrong
“I think there's sort of like a meme that's going around of people saying, we've plateaued, nothing's coming, it's slowing down. I actually really think that's wrong.”
Gomez: AI reasoning and planning capabilities will soon hit production
“For the past year, year plus, folks have been focusing on that, and it will be ready for production. And so we'll see that come out, and I think that will be a big change in terms of capability.”
Aidan Gomez: Chatbots Will Not Replace Dating or Cause Birth Rates to Plummet
“There's no world where suddenly we all start dating chatbots and, you know, human birth rates plummet. I don't think that happens, right?”
Aidan Gomez: Enterprise Buyers Will Not Buy Major Purchases From AI Bots
“If I'm getting sold to by a bot, I'm not buying. It's that simple. I don't want to talk to a machine. I like for certain like simple purchases, maybe, but for the purchases that count, The ones that matter to me and my company. I would want a human accountable…”
Aidan Gomez: AI Will Not Cause Mass Unemployment
“The fears around displacement and replacement, both on the consumer side, where we're all gonna get addicted to chatting to these chatbots, and on the workplace, the end of work, you know, there's gonna be mass unemployment. I can't see that happening.”
Aidan Gomez: Cheap, robust humanoid robotics will be cracked within 5 to 10 years
“Soon, someone's gonna crack the nut of general purpose humanoid robotics that are cheap and robust. And so that will be a, that'll be a big shift. I don't know if that comes in the next five years or 10 years. But it's going to be somewhere in that range.”
Gomez: AI models are hyper-sensitive to single bad data examples
“Like a single bad example, right, amongst, like, billions. Like, it's so sensitive. Like, it is a bit surreal how sensitive The models are to their data. Everyone underrates it.”
Aidan Gomez: Rejects Hinton's AI doomsday view, aligns with LeCun
“No, I'm way more aligned with Yan and his beliefs about AI. So Jeff is like very, you know, doomsday pilled and thinks that this technology is gonna destroy the world. Yan is much more optimistic, and I'm aligned in that direction.”
Gomez: Yann LeCun has become an 'Elon reply guy'
“I think that Unfortunately, Jan has kind of become an Elon reply guy.”
Gomez: European culture is hostile towards tech and obsessed with regulation
“The culture is just hostile towards tech. It's hostile. Like the solution to tech is regulation in the European mind.”
Gomez: Shifting Europe's anti-tech regulation mindset might take a decade
“I think there's pressure to change though, and France is becoming much more ambitious and making a lot of noise on the European stage as well as the global stage about we need to be more progressive. It might take a decade though.”
Shorts cut from this episode
▶ Why Ilya Sutskever is an AI genius 🧠 · 20VC with Harry Steb (@34:39)
▶ OpenAI’s new strategy 👀 · 20VC with Harry Stebbings (@35:45)
▶ AI smarter than all humans 🤖 · 20VC with Harry Stebbings (@24:27)
▶ Why UK beats EU for Startups 🇬🇧 · 20VC with Harry Stebbing (@59:44)
▶ Hidden skill of entrepreneurs 🤫 · 20VC with Harry Stebbings (@2:56)
▶ The future of education 🤖 · 20VC with Harry Stebbings (@0:04)