Apr 10, 2025 · 1h 31m · lennys-podcast
OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
OpenAI Chief Product Officer Kevin Weil joins Lenny Rachitsky to discuss how artificial intelligence is fundamentally transforming software development, product management, and startup strategy. Weil shares tactical insights on writing model evaluations, adopting 'vibe coding,' architecting model ensembles, and building on the frontier of rapid model evolution.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 25.1% of the talking time here. How this is scored →
Lenny as informed peer 5.0 Guest teaching 4.9 Guest disagreement 1.1 Lenny pushing back 1.1
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:27:00 Dismantling the prompt engineering myth
Kevin strongly dismisses the notion that prompt engineering is a permanent long-term skill, asserting that tooling must eliminate prompting friction entirely for AI to succeed.
Hardest push from Lenny ▶ 32:50 Questioning OpenAI's coding model parityLenny directly confronts Kevin on why Anthropic's Claude 3.5 Sonnet holds an edge in developer coding workflows, demanding an explanation for OpenAI's positioning.
Biggest teaching moment ▶ 17:20 Fuzzy probabilistic inputs versus deterministic systemsKevin educates Lenny on fundamental architectural shifts in product building, contrasting reliable deterministic databases with non-deterministic probabilistic LLM behavior.
Lenny holds their own ▶ 1:03:15 Synthesizing model ensembles with organizational structuresLenny demonstrates deep product expertise by taking Kevin's model ensemble concept and formulating an apt metaphor comparing it to specialized corporate personnel.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| Viral Image Generation Launch and Internal Dogfooding Culture | 4 | 2 | 1 | 1 | Lenny opens by referencing the viral reception of OpenAI's new image generation model and connects it to consumer adoption trends. Kevin elaborates on internal dogfooding dynamics, drawing historical parallels to shipping Instagram Stories. | |
| Redefining AI, AGI, and Tech Normalization | 4 | 5 | 2 | 1 | Lenny jokingly presses Kevin on an exact AGI release timeline. Kevin reframes the definition of AI as whatever technology hasn't been mastered yet, illustrating how cutting-edge breakthroughs rapidly normalize into mundane background algorithms. | |
| Kevin Weil's Recruitment Journey to OpenAI | 3 | 1 | 0 | 1 | Lenny prompts Kevin to share behind-the-scenes recruiting anecdotes regarding his transition into OpenAI. Kevin shares an amusing narrative about nine days of silence from Sam Altman and the executive team before landing the CPO offer. | |
| Navigating Non-Deterministic AI Product Development | 5 | 6 | 1 | 1 | Lenny inquires about the core operational differences between managing traditional SaaS and working at the frontier of AI. Kevin explains the shift from deterministic inputs and outputs to probabilistic models requiring probabilistic product frameworks. | |
| The Crucial Role of Evals in AI Product Building | 6 | 7 | 1 | 2 | Lenny cites Kevin's prior remarks regarding evals becoming a baseline product management competency and relates it to traditional unit tests. Kevin expands deeply on evals as continuous learning benchmarks using OpenAI's Deep Research feature as a prime case study. | |
| Startup Moats and OpenAI's API Ecosystem | 6 | 6 | 1 | 2 | Lenny directly asks where foundational model companies will avoid expanding to give early-stage AI startups defensible moats. Kevin quotes Ev Williams and clarifies that OpenAI cannot realistically tackle vertical-specific private data silos, favoring an API ecosystem approach. | |
| Model Benchmarks, Competition, and ChatGPT's Market Edge | 6 | 5 | 2 | 2 | Lenny poses a candid question about why competitor models like Anthropic's Claude 3.5 Sonnet excel at coding. Kevin acknowledges Anthropic's achievements while emphasizing that OpenAI's one-stop multimodal platform sustains consumer mindshare and engagement. | |
| Designing Intuitive Reasoning and Chain-of-Thought UX | 5 | 6 | 1 | 1 | Lenny asks for counterintuitive product design takeaways from building frontier AI systems. Kevin breaks down the user experience considerations of reasoning models, explaining why summarized intermediate thoughts mirror natural human problem-solving pauses better than raw verbosity. | |
| Why Natural Chat Is the Ultimate Universal Interface | 6 | 5 | 2 | 1 | Lenny recalls Kevin's contrarian defense of chat interfaces against widespread industry critique. Kevin argues that open-ended conversational dialogue represents the most flexible, universal human communication protocol available for high-intelligence models. | |
| Sponsor Interview: One Schema File Feeds Launch | 2 | 2 | 0 | 0 | Lenny conducts a brief sponsored segment with OneSchema founder Christina Gilbert detailing their automated CSV integration and data validation tooling. | |
| Hiring High-Agency Product Managers Under Extreme Ambiguity | 5 | 5 | 1 | 1 | Lenny asks about hiring profiles and organizational structure for product management inside a research-centric organization. Kevin highlights high agency, comfort with ambiguity, decisive leadership, and keeping PM headcount deliberately lean relative to engineering. | |
| Embracing Vibe Coding and Prototyping Workflows | 6 | 4 | 1 | 1 | Lenny brings up predictions about automated programming and asks how OpenAI staff leverage AI internally. Kevin explains Andrej Karpathy's concept of vibe coding, expressing that product leaders should build rapid functional prototypes instead of relying solely on static mockups. | |
| Model Ensembles and Specialized Fine-Tuned Architectures | 7 | 6 | 1 | 1 | Lenny brings up real-world developer tools using multi-model setups, prompting Kevin to explain OpenAI's internal use of model ensembles and fine-tuned specialized architectures. Lenny builds on the insight by comparing model ensembles to varied human teams. | |
| Parenting in an AI World and Global AI Tutoring | 4 | 6 | 1 | 1 | Lenny reads an audience question regarding educating children in an AI-dominated economy. Kevin discusses raising curious and self-confident children while advocating for the immense, unrealized potential of globally accessible personalized AI tutors. | |
| Technological Optimism, Societal Transition, and Creative Tools | 5 | 5 | 1 | 1 | Lenny probes societal concerns about AI labor displacement and changes in creative fields. Kevin presents an optimistic perspective, sharing how Hollywood directors use video generation models like Sora to rapidly explore hundreds of pre-production cutscene concepts. | |
| Exponential Progress, Cost Reductions, and Model Maximalism | 6 | 6 | 1 | 1 | Lenny highlights rapid release cycles across foundation models. Kevin emphasizes model maximalism and the steep exponential decline in inference costs, noting that today's AI is the worst version users will ever encounter. | |
| Reflections and Lessons from Facebook's Libra Project | 5 | 6 | 2 | 1 | Lenny asks Kevin to reflect on leading the Libra cryptocurrency initiative at Facebook. Kevin describes the project as the biggest disappointment of his career, candidly breaking down missteps in regulatory timing, scope packaging, and corporate reputation hurdles. |
Statements from this episode (29)
Weil: Teams should question social products that fail internal dogfooding
“Especially social things because you have a very tight network as a company socially. So you know each other and your experts in your product, hopefully. And so there's some sense in which if you're doing something social and it's not taking off internally, yo…”
Weil: OpenAI image model understands complex multi-image spatial instructions
“The model is, is really capable at emulating style or understanding what, you know, it's very good at instruction following. That's actually something that I think people, I'm starting to see people discover with it, but you can do very complex things. You can…”
Weil: AI gives computers novel capabilities every two months
“Every two months, computers can do something they've never been able to do before, and you need to completely think differently about what you're doing.”
Weil: Model accuracy levels dictate fundamentally different product designs
“If the model gets it right, 60% of the time, you build a very different product than if the model gets it right, 95% of the time, versus if the model gets it right, 99.5% of the time. And so there's also something that you have to get really into the weeds on …”
Weil: Future AI will pair broad foundation models with custom evals
“I think the future is really gonna be incredibly smart, broad based models. That are fine tuned and tailored with company specific or use case specific data so that they perform really well on company specific or use case specific things. And you're going to m…”
Weil: 3 million developers currently use OpenAI's API
“We have three million developers using our API.”
Weil: OpenAI will not build industry-specific vertical AI products
“There are immense opportunities in every industry and every vertical in the world to go build AI based products that improve upon the state of the art. And there's just no way we could ever do that ourselves. We don't want to, we couldn't, if we did want to.”
Weil: Multi-month product roadmaps in AI are pointless
“So that's, I think, just expecting that you're going to be super agile and that there's no sense writing a three month roadmap, let alone a year long roadmap, because the technology is changing underneath you so quickly.”
Weil: OpenAI minimizes scaffolding because rapid advances erase model limits
“We don't spend that much time building scaffolding around the parts that don't match that because our general mindset is in two months, there's going to be a better model and it's going to blow away whatever, you know, the current set of limitations are.”
Weil: Developers Building on the Edge of Model Limits Should Persist
“If you're building And the product that you're building is kind of right on the edge of the capabilities of the models. Keep going, because you're doing something right. Because you give it another couple months, and the models are going to be great. And sudde…”
Weil admits OpenAI no longer holds a massive 12-month lead
“It used to be that OpenAI had this, like, massive model lead, you know, 12 months or something ahead of everybody else. That's not true anymore. You know, I like to think we still have a lead. I'd argue that we do, but it's certainly not a massive one.”
Weil: AI product UX can be designed by modeling human behavior
“One of the things that's been funny for me is the extent to which you can kind of reason when you're trying to figure out how some products should work with AI, you can often, or even why some AI thing happens to be true. You can often reason about it the way …”
Weil: Model ensembles attacking a problem together yield better thinking
“You get better thinking sometimes out of a group of models that all try and attack the same problem, and then you have a model that's looking at all their outputs, and integrating it, and then giving you a single answer at the end.”
Weil: Specialized UIs suit narrow AI tasks, but chat is the baseline catch-all
“By the way, I think there are like, it's not that it's only chat either, like there are, if you have high volume use cases where they're more prescribed, and the, you don't actually need the full generality, there are many use cases where it's better to have s…”
Weil: OpenAI should never become a pure product company
“OpenAI should never be a pure product company. We need to be both a world-class research company and a world-class product company. And the two need to really work together.”
Weil: OpenAI operates with only about 25 product managers
“Not that many, actually. I don't know. 25. Maybe it's a little more than that”
Weil: Organizations should stay PM-light to prevent micromanagement
“Too many PMs causes problems. You know, we'll like fill the world with decks and ideas versus execution. So I think that the, I think it's a good thing when you have a PM that has that is working with maybe slightly too many engineers, because it means that th…”
Weil: OpenAI struggles with junior PMs due to extreme ambiguity
“And we have trouble sometimes with more junior PMs because of this, because it's just not the place where someone is going to come in and say, okay, you know, here's, here's the landscape. Here is your area. I want you to go do this thing. And that's what you …”
Weil: Product teams should vibe-code interactive prototypes instead of using Figma
“Why shouldn't we be, like, vibe coding demos right, left, and center? Like, instead of showing stuff in, like, Figma, we should be showing prototypes that people are vibe coding, you know, over the course of 30 minutes to illustrate proofs of concept and to ex…”
Weil: OpenAI's Chief People Officer vibe-coded an internal company tool
“Our, actually, our chief people officer, Julia, Was telling me the other day, she vibe coded an internal tool that she had at a previous job that she really wanted to have here at open AI. And she opened, I don't know, windsurf or something and vibe coded it.”
Weil predicts ML engineers will soon embed in every product team
“And so I think you're going to want sort of quasi researcher machine learning engineer types as part of pretty much every team because fine tuning a model is just going to be part of the core workflow for building most products.”
Weil: OpenAI manages 400 million weekly users with 30-40 support staff
“With 400 plus weekly 400 plus million weekly active users, we get, you know, a lot of inbound tickets, right? I don't know how many customer support folks we have, but it's not very many. 3040, I'm not sure. Way, way smaller than you would have at any comparab…”
Weil: Coding skills will remain relevant for a long time
“I think, you know, things like coding skills are going to be relevant for a long time.”
Weil: Current AI models are already good enough for global tutoring
“The models are good enough to do it now, and every, every study out there that's ever been done seems to show that when you have, you know, classrooms is still, classroom, like education is still important, but when you combine that with personalized tutoring,…”
Weil: OpenAI aims to release new o-series models every 3-4 months
“And now with this O series of reasoning models, we're moving even faster. We're like every roughly, you know, three months, maybe four months, there's a new O-series model, and each of them is a step up in, in capability, and so the capabilities of these model…”
Weil: GPT-4o mini is 100x cheaper via API than GPT-3.5
“The original, I think the original, I don't know, what was it, GPT 3.5 or something, was like a hundred x the cost of GPT-Foro Mini today. In, in the API. So a couple years, you've gone down two orders of magnitude in in cost. For much more intelligence.”
Weil calls Facebook's Libra the biggest disappointment of his career
“Honestly, Libra is probably the biggest disappointment of my career.”
Weil: Aptos and Mysten Labs are built on Facebook's Libra technology
“There are a couple of current blockchains that are built on the tech because the tech was open sourced from the beginning. Aptos and Mistin are two companies that are built off of this tech, so, you know, at least the, all of the work that we did not die, but,…”
Weil: Prompt engineering will become obsolete as AI matures
“I want to kill the idea that you have to be a good prompt engineer. I think if we do our jobs, that stops being true. You know, it's just one of those, like, sharp edges of models that experts can learn, but then you just, over time, you shouldn't need to know…”