Jul 26, 2026 · 1h 33m · lennys-podcast
Why AI is going vertical (again) | Dianne Penn (Anthropic)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Anthropic's Head of Product for AI Research and Labs, Dianne Penn, joins Lenny Rachitsky to discuss how frontier models, purpose-built harnesses like Claude Code, and eval-driven product management are transforming software development. She shares insider lessons on Anthropic's rapid experimentation culture, navigating exponential capability jumps, and why human judgment remains essential in an AI-driven world.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 30.2% of the talking time here. How this is scored →
Lenny as informed peer 4.1 Guest teaching 4.0 Guest disagreement 0.1 Lenny pushing back 0.3
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 20:37 Reframing Garry Tan's token spend thesis
Dianne gently challenges the premise of spending $100k on tokens as a benchmark, arguing that token spend is merely an input while collaborative experimentation is the actual output.
Hardest push from Lenny ▶ 48:02 Pushing back on whether PRDs are deadLenny presses Dianne on whether PRDs are obsolete in favor of evals, prompting her to clarify where comprehensive written specs remain indispensable.
Biggest teaching moment ▶ 28:40 Deconstructing vague feedback into researcher-actionable evalsDianne gives Lenny an operational masterclass on why telling researchers Claude hallucinated is useless, demonstrating how PMs must break failures down into tool-use, search synthesis, or alignment issues.
Lenny holds their own ▶ 46:42 Comparing eval-driven development to TDDLenny demonstrates sharp domain expertise by synthesizing Dianne's eval framework into the classic software paradigm of test-driven development.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| Episode Preview: Key Themes and High-Velocity AI Insights | 1 | 0 | 0 | 0 | Lenny introduces Dianne Penn and previews the episode themes alongside brief promotional snippets from the conversation. | |
| Anthropic's Humble Beginnings and the Golden Gate Claude Inflection | 3 | 4 | 0 | 1 | Lenny recalls Anthropic's early days competing against OpenAI, while Dianne explains the culture and the Golden Gate Claude experiment. | |
| Sponsor Message: WorkOS Enterprise Solutions | 4 | 5 | 0 | 0 | Following an ad read, Lenny asks about inflection points, and Dianne breaks down how Opus 3 and long-form coding differentiation established Anthropic's position. | |
| The Synergy of Opus 4.5 and Claude Code | 5 | 4 | 0 | 0 | Lenny notes Dario Amodei's accurate coding predictions and the exponential curve, prompting Dianne to discuss adaptability and first-principles reasoning. | |
| Discontinuous Capabilities, Scaling Laws, and Product Overhang | 5 | 5 | 1 | 1 | Lenny summarizes model adaptability, and Dianne refines the explanation using scaling law graphs and discontinuous emergent capabilities. | |
| Token Spending, Communal Experimentation, and Working in Public | 4 | 4 | 1 | 0 | Lenny brings up Garry Tan's token-spending thesis, which Dianne gently reframes from raw spend input to communal experimentation output. | |
| Anthropic Labs: Incubating Discontinuous Bets Outside the Core Roadmap | 4 | 4 | 0 | 0 | Lenny inquires about Anthropic Labs' operational structure, and Dianne outlines how small pods incubate high-risk discontinuous bets. | |
| Inside the AI Research Loop: Translating User Feedback into Evals | 4 | 6 | 0 | 0 | Lenny asks what AI researchers do all day, and Dianne provides a detailed operational breakdown of translating vague user complaints into rigorous evals. | |
| Traits of World-Class Researchers and Designing for Forward Compatibility | 4 | 4 | 0 | 0 | Lenny asks about researcher traits, and Dianne describes how forward compatibility and first-principles thinking define top AI talent. | |
| Frontier Model Safety, Safeguard Packages, and Responsible Deployment | 4 | 3 | 1 | 1 | Lenny highlights increasing safety scrutiny and restricted releases; Dianne sets boundaries regarding policy but explains fallback UX systems. | |
| Sponsor Message: Mercury Banking with Command AI Operator | 5 | 6 | 0 | 0 | After an ad break, Dianne explains how PM hiring criteria remained steady while the core PM artifact shifted from PRDs to evals. | |
| Case Study: Resolving JSON Output Reliability Through Custom Evals | 5 | 5 | 0 | 0 | Dianne illustrates an eval case study concerning JSON output compliance, which Lenny compares to test-driven development. | |
| Balancing Evals and PRDs for Large-Scale Alignment and Vision | 4 | 4 | 0 | 1 | Lenny checks if PRDs are truly dead; Dianne confirms they remain crucial for broad multi-team alignment and exploratory, ambiguous capabilities. | |
| Hands-On AI Leadership: Why Managers Must Build and Ship | 4 | 4 | 0 | 0 | Dianne emphasizes that engineering and product managers must actively build and ship to evaluate AI output effectively. | |
| Collaborative Discovery and Deep Immersion in Specific AI Use Cases | 5 | 3 | 0 | 0 | Lenny and Dianne discuss community survey insights showing that going deep on narrow use cases yields more satisfaction than superficial tinkering. | |
| Augmenting EQ: Tactical Claude Workflows for Coaching and Thinking | 4 | 4 | 0 | 1 | Dianne shares how she uses Claude skills for Crucial Conversations coaching, and addresses Lenny's query about cognitive reliance on AI. | |
| The Power of Claude's Constitution: Why Pushback Drives Quality | 5 | 4 | 0 | 0 | Lenny notes Claude's distinctive conversational personality, and Dianne explains how constitutional alignment creates a genuine thinking partner through pushback. | |
| The Jagged Frontier of AI Writing and Future Improvements | 5 | 4 | 0 | 1 | Lenny asks why LLMs struggle with natural prose writing; Dianne explains the jagged capability frontier and prioritizing agentic milestones. | |
| Human Judgment, Lifelong Learning, and Developing an Inner Voice | 4 | 4 | 0 | 0 | Dianne highlights enduring human strengths like judgment and persistence, relating them to parenting strategies in an AI-dominated era. | |
| Sustaining High Performance: The Hive Mind and Team Resilience | 4 | 4 | 0 | 0 | Dianne describes how the Anthropic team maintains resilience through collective ownership, shared coverage, and an ego-free hive mind. | |
| Closing Reflections: The Enduring Need for User-Centric PMs | 5 | 4 | 0 | 0 | Dianne summarizes why deeply user-centric PMs will remain vital, which Lenny enthusiastically validates. | |
| Lightning Round: Books, Culture, and Trading Floor Lessons | 3 | 3 | 0 | 0 | During the lightning round, Dianne shares recommendations and reflects on how her bond trading desk experience taught her to champion conviction and truth over hierarchy. |
Statements from this episode (26)
Penn: Anthropic had five product engineers and one API engineer in 2023
“So I joined in 20, 23. Like you said, we had five product engineers. There was one engineer for the entirety of our API business, if you believe.”
Penn: Golden Gate Claude was Anthropic's hidden product inflection point
“That to me was like one of those, like maybe hidden inflection points of we were starting to find our identity, that we could build products, build experiences that were different for what our competitors had seen, what was already out there.”
Penn: Anthropic had fewer than 200 employees while building Claude 3 Opus
“Definitely when we were training and testing Opus three, I think that was the moment when the company, I think we were less than 200 people still at that point.”
Penn: Optimizing Claude 3 Opus for long-form coding drove Anthropic's early differentiation
“People are starting to use code these models, not just for code, not just like code autocomplete, but actually writing long form code. And it's that an opportunity for us to train, you know, Opus three to be better at. And it ended up being a relatively smalle…”
Penn: Users need frontier product harnesses to feel frontier model magic
“One thing we say a lot on the team is you need frontier products in order to have frontier models and for people to feel the magic. Of frontier models.”
Penn: Opus 4.5 and Claude Code depended on each other for breakout success
“I think Opus Four Five wouldn't have had that moment without a product like Cloud Code. And Cloud Code, I think, wouldn't have had that type of adoption accelerated without Opus Four Five.”
Penn: Scaling loss is smooth, but AI capabilities emerge discontinuously
“As you add in more compute and data, what's called loss, aka the loss from next token prediction goes down. And so it's a very smooth linear curve of, like, the models get more intelligent as you scale them up. What's actually also interesting in that paper is…”
Penn: Anthropic's Opus and Fable models still have product overhang
“I think there's like product overhang and user overhang, like to maybe put it in our PM language, even on today's models. And I think there's like a lot that we could be exploring on like our current opuses and definitely with like Fable, for example”
Penn: Anthropic Labs incubated Claude Code, Skills, Claude Design, and MCP
“Things like cloud code things like skills, and most recently cloud design, MCP.”
Penn: Anthropic discussed computer use for Claude at founding
“I think even at the founding of the company, researchers were talking about how do we get clawed to you? Use a computer. How do we get AI to, like, navigate a screen, right?”
Penn: Telling AI researchers Claude hallucinated is unactionable without failure-mode breakdowns
“If you bring that to a researcher and you say, please fix Claude from being hallucinated, it's not very actionable. And so part of the time of the team is understanding, okay, what's the trajectory of why that user gave that feedback?”
Anthropic Research Leaders Personally Inspect Training Runs and Evals
“Our like leadership, our chief scientists, our heads of like fine tuning and like RL folks are actually really close to the training runs and actually look at things like how the training run is going, evals, looking at the underlying data.”
Penn: Design AI products for forward compatibility with future Claude models
“One thing I asked the team frequently or how I think about when we're building a product is let's say Claude eight comes around. What do you, what changes in what users do? And then what should, what does that mean for how you're building today? Is it going to…”
Penn: Anthropic Kept Research PM Hiring Process Unchanged for Three Years
“We actually on my team have not changed our hiring loop for three years now. So what we actually look for and the traits and how we evaluate generalists like PNs, generalists like research product managers have actually been the same.”
Penn: Evals Have Replaced Traditional PRDs in AI Product Management
“We actually have a saying on the team of evals are the new PRDs. Cause in order to deliver that user value it's not that exact artifact that people used to write in the last like one to two decades. It's a new way of working.”
Penn: AI PMs Must Sweat Tokens as Much as Pixels
“Here, you have to sweat the tokens as much as you sweat the pixels. And so one activity we have on the team is reading the transcripts. And understanding, ah, what was the trajectories that failed?”
Penn: 80% of early Claude instruction failures were invalid JSON outputs
“And what I saw was something like 80% of what people meant in the early days for this failure was Claude would not write the right JSON.”
Penn: Claude scores 99.9% to 100% on JSON schema formatting evals
“When we have versions of Claude, we actually run that eval and just check. I think at this point it's always a hundred percent or like 99.9. And so it's no longer a pain point.”
Anthropic creates a PRD for every AI model it develops
“So when we do have a model, we actually, for every model, we do have a PRD. Less necessarily for our researchers, but more for our growing product surfaces, for our engineering teams, for our stakeholders like legal and safety and others as just a source of tr…”
Penn: Built Claude skill using Crucial Conversations to coach management conversations
“I don't think it's necessarily just about raising the IQ of like experiences we build, but also I use it a lot and actually like prepping for how to have better conversations in the moment during like crucial conversations. So I love that book. And so I actual…”
Penn: Useful AI models push back on unformed ideas rather than agree
“What you don't want is like a AI that just agrees with you, right? What you want is this technology to actually augment and grow and like get to a better outcome. And so sometimes it's Having Claude push back makes me better, and so that's great. Like a cowork…”
Penn: Anthropic uses research Opus models to determine future Claude pricing
“So I've used Claude to help with things like, are we making the right pricing decision on the next version of Claude? It's a little bit meta, but using a research version of Opus, asking it to figure out how is your price and being able to come out with better…”
Penn: Human judgment will stay critical as AI lacks human experience
“I think judgment is one, is an area where it's accumulation of so much nuance and so much experience, and these systems haven't experienced as much as humans have. And so I think that hard earned, like, judgment Is a area for product leaders and just generally…”
Penn: AI in biology and life sciences is only at foot of exponential
“I think, you know, software engineering has been really transformed by AI. I think there's areas like biology, life sciences. These are all things that we're just kind of at like the foot of the exponential on. Like, maybe software engineering were on the expo…”
Penn: Highly capable AI models increase the need for user-centric PMs
“I think fundamentally, you didn't ask me this, but there is this question in the community of, do we still need PMs when the models are still capable, when engineers are leaning in I think the role of people who are user centric, who go into the details of und…”
Penn: Anthropic UI ratings and sales feedback route directly to research PMs
“If you thumbs up or thumbs down on any of our product surfaces, if you contact your salesperson with feedback about the model, it will make its way to me. We actually, with every, like, research model, I actually get pretty close into understanding favorabilit…”