Oct 2, 2024 · 1h 16m · news
Bret Taylor: Why Pre-Training is for Morons & Companies Will Build Their Own Software | E1209 · 20VC with Harry Stebbings
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Tech veteran Bret Taylor shares critical insights on the AI landscape, detailing the shift from deterministic to probabilistic software, the capital inefficiencies of model pre-training, and how his platform, Sierra, is shaping the future of conversational AI agents.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Harry holds 16.3% of the talking time here. How this is scored →
Harry as informed peer 3.8 Guest teaching 3.8 Guest disagreement 1.6 Harry pushing back 2.7
speaking balance: gold is Harry, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 24:00 Bret's stark critique of startup pre-training
Bret forcefully rejects the premise of non-AGI startups building pre-trained models, calling it nonsensical capital burning equivalent to hand-building data centers.
Hardest push from Harry ▶ 10:06 Harry challenging the dot-com bubble comparisonHarry directly refuses Bret's dot-com framing by citing xAI's $18B valuation and explaining that 1998 rounds were priced with vastly different dilution and multiple dynamics.
Biggest teaching moment ▶ 13:40 Bret reframing vertical software survival using cloud historyBret corrects the host's premise that foundation models will absorb software applications, educating him on total cost of ownership and why CIOs buy turnkey SaaS instead of building from a bag of floating point numbers.
Harry holds his own ▶ 17:51 Harry presenting market data on AI services profitabilityHarry counters Bret's application-centric view by bringing specific industry data showing IT services firms posting billions in profit and outearning OpenAI.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Harry as informed peer | Guest teaching | Guest disagreement | Harry pushing back | Why |
|---|---|---|---|---|---|---|
| Bret Taylor's Path to Software Obsession | 1 | 1 | 0 | 0 | Harry asks a friendly background question about whether Bret knew he would be successful as a child. Bret shares his personal journey from gas station attendant to making local business websites and attending Stanford. | |
| Can Entrepreneurship and Leadership Be Learned? | 2 | 2 | 0 | 1 | Harry asks if entrepreneurship is innate or learned, drawing on his own experience doing 3,000 podcast interviews. Bret reframes leadership as a craft akin to military training rather than an innate talent. | |
| The AI Bubble and Dot-Com Analogies | 5 | 4 | 2 | 6 | Harry directly challenges Bret's dot-com bubble analogy by highlighting that modern mega-rounds like xAI at $18B involve vastly higher valuations and lower return multiples than 1998 startups. Bret acknowledges Harry's VC lens while maintaining his perspective on broader economic impact. | |
| Why Companies Choose to Buy Rather Than Build Software | 3 | 5 | 2 | 2 | Harry asks if advanced models will subsume vertical software. Bret dismantles the premise using a cloud market analogy, explaining that software is like a lawn that requires upkeep and companies prefer buying ready solutions over building from raw models. | |
| The Rise of AI Consulting and Change Management | 6 | 4 | 2 | 4 | Harry presents his Twitter thesis that AI consulting services will be the biggest financial winners, citing firms generating billions in profits. Bret agrees on short-term implementation demand but clarifies that long-term value lies in operational change management. | |
| The Categorization and Commoditization of AI Models | 3 | 5 | 6 | 2 | Bret forcefully criticizes startups that pre-train models, calling it capital burning and nonsensical unless operating as an AGI research lab. He categorizes models into foundation versus frontier tiers to explain commoditization. | |
| The Three Pillars of Progress Toward AGI | 3 | 5 | 2 | 3 | Harry presses on whether model improvements face diminishing returns after GPT-4. Bret breaks down AI progress into three distinct pillars—data, compute, and algorithms—explaining why progress across all three makes a total plateau unlikely. | |
| Reconciling the Pursuit of AGI with Real-World Products | 5 | 4 | 3 | 5 | Harry challenges OpenAI's dual priorities of pursuing AGI while launching enterprise and consumer products, contrasting it with Perplexity's single-minded focus. Bret reframes product distribution as the primary vessel for delivering AGI benefits to humanity. | |
| Proprietary Knowledge and Sustainable AI Business Models | 4 | 4 | 1 | 2 | Harry inquires about sustainable business models given heavy training and inference costs. Bret explains why inference costs are dropping rapidly due to distillation and hardware efficiencies, following a trend similar to Moore's Law. | |
| Hyperscaler Capital and Meta's Open-Source Strategy | 5 | 4 | 2 | 4 | Harry points out Meta's lack of a cloud business cash cow compared to Google and Amazon to fund capex. Bret praises Zuckerberg's open-source strategy with Llama 3.1, drawing parallels to open-source infrastructure like Postgres and Linux. | |
| Supercomputers, Sunk Costs, and Market Consolidation | 5 | 3 | 2 | 4 | Harry asks if supercomputer capex represents a sunk cost trap like the Manhattan Project and notes recent consolidation among model builders. Bret validates the bold capex investments by mega-caps while agreeing that pre-training startups face inevitable consolidation. | |
| Building Branded Customer Agents: Sierra's Core Vision | 2 | 4 | 0 | 1 | Bret outlines Sierra's vision for branded conversational AI agents. He draws a historical analogy comparing the paradigm shift from physical BlackBerry keyboards to multi-touch iPhones with the shift to conversational software. | |
| Smartphones as Mediators of Conversational Interfaces | 4 | 4 | 2 | 3 | Harry questions whether smartphones will be replaced by smart glasses like Meta's Ray-Bans. Bret tempers this expectation, highlighting 15 years of failed hardware attempts to unseat smartphones and predicting phones will remain the core computing anchor. | |
| Shift in Software Design: Goals and Guardrails | 3 | 5 | 1 | 2 | Bret explains the key engineering hurdle at Sierra: transitioning software design from fixed rule sets to goals and guardrails due to the probabilistic nature of generative AI. | |
| Navigating the Trade-Off Between Agency and Control | 4 | 5 | 2 | 4 | Harry questions whether agents are restricted to low-risk tasks like pizza refunds to avoid high-stakes errors. Bret details the trade-off dial between agency and control, noting enterprise customers use agents for core revenue and churn management. | |
| Probabilistic Software and Human Error Management | 4 | 4 | 1 | 1 | Harry notes that human call center workers also make mistakes and hallucinate. Bret agrees, explaining that software engineers must shift away from expecting deterministic perfection and instead adopt human operational risk controls for AI. | |
| Veracity of Information and Iterative AGI Safety | 5 | 3 | 1 | 3 | Harry raises concerns regarding media verification and quotes Princeton professor Arvin Narayanan on widespread skepticism. Bret emphasizes responsible iterative deployment and predicts AI verification tools will act as security defenses. | |
| Fundraising Philosophy and the Value of Great Boards | 4 | 3 | 0 | 1 | Harry conducts a quickfire round covering fundraising, board governance, and Bret's experiences with Mark Zuckerberg and Marc Benioff. Bret details why he raises venture capital to ensure board accountability. |
Statements from this episode (35)
Bret Taylor: AI pre-training outside AGI labs wastes capital
“Unless you are an AGI research lab, doing pre-training on a model I believe is just burning capital.”
Bret Taylor quit his gas station job after a $400 web gig
“I was getting paid four dollars and 25 cents at the gas station an hour, which was minimum wage at the time. And ended up getting paid 400 dollars for the website. So I quit the gas station job the next day and ended up making websites for a lot of local busin…”
Taylor: High anxiety makes succeeding as an entrepreneur difficult
“It's hard to be an entrepreneur if you're prone to anxiety because everything's on fire all the time. You know, that's just the nature of the business.”
Taylor: Top enterprise software companies are founded by older entrepreneurs
“A lot of the greater enterprise software companies were started by entrepreneurs. Entrepreneurs later in their career which is, I think really interesting as well.”
Taylor: AI bubble will rhyme with dot-com era, where excess was justified
“And I think the AI bubble will rhyme with the dot com bubble. And I believe with the benefit of hindsight, Most of the excess of the.com bubble might have been justified.”
Taylor: AI will yield a trillion-dollar consumer giant and 10+ public companies
“We will look back and laugh at some of the excess, but I'm confident we will have, you know, a brand defining and likely trillion dollar consumer company come out of this. 10 plus enterprise software companies that are enduring, you know, public companies comi…”
Bret Taylor: Commercial AI market will mirror cloud software structure
“I actually think the AI market commercially will play out like the cloud market did over the past 20 years.”
Bret Taylor: Sierra fine-tunes third-party models rather than pre-training
“So at Sierra, which is my company, we make a solution. We're not doing pre-training where, you know, we're fine tuning other people's models to build this solution.”
Bret Taylor: Smart AI models won't alter enterprise software buying habits
“And so I think this idea that somehow the way the world wants to buy software will change because these models are really smart. Doesn't resonate with me.”
Stebbings: AI services companies will be AI's biggest winners in 3-5 years
“I think AI services companies over the next three to five years will actually be the biggest winners in AI.”
Taylor: Software companies are inherently bad at organizational change management
“I think one thing that software companies have always been bad at for good reason. I don't think it's necessarily what we do is actually helping companies manage the adoption of this technology.”
Taylor: Enterprise spending on AI professional services will drop as software matures
“I think there's probably some short term professional services spend that reflects the lack of the maturity of the AI applications market right now. You know, and I think that when there are solutions like Sierra and others for specific domains available, you …”
Bret Taylor: Companies needing standard models should simply fine-tune Llama or Mistral
“You know, I think that in that market probably if you need a model like that, you should download llama. That's the answer. It's like, you don't need much of a cheat sheet on that, you know, and or maybe Mr. All, but pick one of the open source models that are…”
Bret Taylor: Iterative deployment is the most responsible path to AGI
“I believe the most responsible way to develop AGI is responsible iterative deployment. And the reason for that is I believe that as you're thinking about things like the societal impact access to this technology and the safety side of AGI as well, that the, Be…”
Taylor: AI progress won't stall because data, compute, and algorithms alternate bottlenecks
“In any one of those, you could probably make a very rational intellectual case that we're gonna hit a wall, but then you have the two others. And I don't think you can make the case for all three that they're all coming up on a wall. And I think like any big s…”
Taylor: ChatGPT is the most important tech product of the past decade
“At least my understanding why it has a sort of a goofy name is it was a research preview that turned out to be the most important product of the past decade, you know, and I think that, you know, one of the things I think about is, wow, what an important mecha…”
Taylor: Most companies should avoid model pre-training and align costs with usage
“Most companies should be applying AI to build solutions and most companies should have relatively modest training costs and most of their costs should be correlated with inference, which should be correlated with revenue and usage of your product.”
Bret Taylor: AI inference costs will decrease following Moore's Law trajectory
“I'm incredibly optimistic that just the cost of running AI will could probably track something like Moore's law.”
Taylor: AI hyperscalers have an imperative to maintain massive capex spending
“I think these companies have an imperative because of the potential impact of AI to spend and, You know, the capex numbers are mind-boggling, but I probably would do the same thing.”
Bret Taylor: Consolidation ahead for AI pre-training companies
“I think we'll see consolidation of companies pre-training their own models you know, and I think that the cost structure of the tools and applications companies are different and perhaps more sustainable.”
Taylor: Conversational AI will replace web and app clicks within five years
“And as a consequence, you know, I think that, you know, if you fast forward four or five years, when you're interacting with any of the consumer brands, you work with your insurance company, your phone company a car rental company, you will probably be having …”
Taylor: Having an AI agent will be essential by 2025
“We like to say like in 1995, the way you existed digitally as a business to have a website in 20, 25, the way you will exist digitally is to have an AI agent.”
Bret Taylor: Conversational AI will become the dominant computer interface
“And my thesis is just like touch screens have come to dominate Our experience with computers because of convenience, you can have a conversation in so many different places. You don't need an instruction manual. I think it will be the main way we work with com…”
Taylor: Smartphones will remain the primary interface for conversational AI
“So in the short term, my intuition is that the combination of a smartphone with Ray-Ban glasses or AirPods or the like probably, you know, meaning you might need to look at your screen less than you do today. But my intuition is because of the prevalence of sm…”
Taylor: Agent-native hardware requires consumer software experiences to lead first
“I have a sense that we will, it will take a little while to see agent native consumer experiences and agent native devices. And the hard part about particularly consumer electronics is you kind of need the consumer experiences to lead a little bit, to have the…”
Bret Taylor: Software is shifting from fixed rules to goals and guardrails
“We like to say software's going from the age of rules to the age of goals and guardrails.”
Taylor: Sierra customers already use AI agents for sales and churn management
“Our customers already are using it for revenue generation, sales subscription churn management for subscription services, things like that.”
Taylor: AI agents face a fundamental trade-off between agency and corporate control
“The more you give an agency, the more it will have empathy and feel delightful, but the less control you'll have over it.”
Taylor: AI will create new job roles like agent engineers and AI architects
“We think that there's the role of an agent engineer who builds these agents on their platform. We think there's a role of an agent AI architect, who's a customer experience leader, whose job is to do the conversation design and shape The behavior of these agen…”
Taylor: Companies must manage AI mistakes operationally rather than eliminating them
“Stop putting, you know, AI software in the bucket of computers and that rule set and how you deal with it to try to get to, you know, five nines of repeatability and say, okay, this is actually gonna be, A really creative, really impactful, much lower cost sol…”
Taylor: AI-generated problems will largely be solved by AI solutions
“I really do believe for most of the problems at AI, there are AI solutions to those problems as well.”
Bret Taylor only pitched Benchmark's Peter Fenton when fundraising for Sierra
“So when we started the company, I just called Peter Fenton, who I've worked with twice before. He's the only person I talked to.”
Taylor: Defining AI companies will be applications, not base models
“The focus on hardware and models and not enough focus on the applications of AI. I think many of the defining companies in AI will be delivering consumer and business solutions that happen to be powered by AI, not just the models themselves.”