Sep 21, 2023 · 1h 12m · lennys-podcast
Becoming evidence-guided | Itamar Gilad (Gmail, YouTube, Microsoft)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Former Google product manager and author Itamar Gilad joins Lenny Rachitsky to explain the evidence-guided product management framework, presenting tools like the GIST model, Confidence Meter, and Metrics Trees to replace opinion-based decision-making. Using case studies from Gmail and Google Plus, Gilad demonstrates how low-cost experimentation and outcome-focused roadmaps de-risk development and drive measurable user value.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 26% of the talking time here. How this is scored →
Lenny as informed peer 5.0 Guest teaching 4.8 Guest disagreement 1.1 Lenny pushing back 1.4
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 44:41 Rejecting the delivery speed versus discovery trade-off
Itamar firmly rejects Lenny's premise that discovery slows down teams, arguing that measuring raw delivery speed is the wrong metric and that opinion-based building causes massive resource waste.
Hardest push from Lenny ▶ 14:23 Lenny challenging whether founder intuition should bypass validationLenny pushes back against pure evidence-guided development by citing Steve Jobs and questioning whether experienced founders should be allowed to run with creative intuition without upfront validation.
Biggest teaching moment ▶ 15:30 Correcting the historical myth of Steve Jobs and the iPhoneItamar dismantles the myth that Steve Jobs simply envisioned the iPhone in isolation, explaining that Jobs originally opposed phone development and only pivoted after teams presented hard multi-touch trial-and-error evidence.
Lenny holds their own ▶ 30:19 Lenny formulating the mathematical model for metrics trees and team designLenny articulates deep product expertise by distilling metrics trees into mathematical formulas that determine lever sensitivity, experimental upside, and organizational team topologies.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| The Failure of Google Plus and Opinion-Based Development | 4 | 3 | 1 | 1 | Lenny prompts Itamar to share the story of Google Plus and Gmail to ground the philosophy of evidence-guided development. Itamar explains how top-down executive conviction led to massive organizational waste on Google Plus while missing mobile opportunities like WhatsApp. Lenny adds historical context about Facebook's internal Defcon-1 reaction. | |
| The Evidence-Guided Success of Gmail's Tabbed Inbox | 4 | 4 | 1 | 1 | Itamar contrasts the Google Plus failure with the bottom-up, evidence-backed success of Gmail's tabbed inbox feature. Lenny jokes about how the promotional tab affects his newsletter while appreciating the massive user reach. The dynamic remains conversational and collaborative. | |
| Evidence-Guided Philosophy and Challenging Executive Intuition | 5 | 7 | 2 | 3 | Lenny plays devil's advocate, citing Steve Jobs and asking if top-down founder intuition sometimes supersedes evidence gathering. Itamar corrects the popular myth of Jobs inventing the iPhone in isolation, explaining it was actually an extensive discovery and trial-and-error process. Itamar then provides tactics for mid-level PMs to challenge senior leadership using hard data. | |
| Warning Signs of an Opinion-Based Organization | 5 | 5 | 1 | 1 | Itamar details the warning signs of opinion-based organizations, including vague output-based goals, missing user metrics, and heavy roadmapping. He introduces the four-layer GIST model, and Lenny asks strategic clarifying questions regarding where product vision and research belong. | |
| GIST Layer 1: Goals, Metrics Trees, and Value Exchange | 6 | 5 | 1 | 1 | Itamar explains the Goals layer and metrics trees, referencing Lenny's published research on North Star metrics while refining the distinction between value created versus value captured. Lenny demonstrates his own domain knowledge by synthesizing metrics trees into mathematical formulas that guide resource allocation and team topologies. | |
| Sponsor Segment: LinkedIn Ads | 6 | 4 | 1 | 1 | After the sponsor read, Itamar explains the Ideas layer using the ICE framework. Lenny notes Sean Ellis's original definition of ease over effort and shares his preference for simple spreadsheet scoring over complex formulas. | |
| Evaluating Strength of Evidence with the Confidence Meter | 5 | 6 | 1 | 1 | Itamar walks through his Confidence Meter tool, showing how self-conviction and thematic trends like AI provide minimal true confidence compared to empirical testing. Lenny connects the tool directly to the practical PM challenge of saying no to executive pet ideas. | |
| Balancing Discovery Speed with Delivery Across Company Stages | 6 | 5 | 2 | 3 | Lenny presses on whether early-stage startups with experienced founders should skip slow validation to maintain delivery speed. Itamar pushes back on the premise, arguing that evidence-guided validation actually shortens time-to-outcome by avoiding the delivery of useless features. Lenny follows up by categorizing which types of companies benefit most from these frameworks. | |
| GIST Layer 3: Steps and the Spectrum of Validation Methods | 4 | 6 | 1 | 1 | Itamar explains the Steps layer, emphasizing low-cost validation techniques like Wizard of Oz tests and fishfooding before writing production code. Lenny expresses genuine interest in the Gmail facade prototype story and the term fishfooding. | |
| GIST Layer 4: Tasks, GIST Boards, and Outcome-Based Roadmaps | 6 | 5 | 1 | 2 | Itamar presents the Tasks layer and GIST boards to bridge the divide between agile developers and planning roadmaps. Lenny helps clarify the implementation details, highlighting that steps are learning milestones rather than delivery deadlines, and inquires about replacing traditional feature roadmaps with outcome-based roadmaps. | |
| Lightning Round and Product Recommendations | 4 | 3 | 0 | 0 | Lenny conducts the standard lightning round covering product book recommendations, learning Spanish via YouTube, synthetic voice cloning tools, and Israeli shawarma. The tone is lighthearted and amicable throughout. |
Statements from this episode (17)
Gilad: Google+ Caused Google to Miss Mobile Social Opportunities Like WhatsApp
“In hindsight, not only did Google bet on the wrong thing, it missed much easier opportunities. So just not far from Google's headquarters, there was WhatsApp. Not very famous in the US, but they actually created massive impact. Hundreds of millions of people w…”
Gilad: Google+ Peak Team Reached Roughly 1,000 Employees
“And I think at its peak, it was about a thousand people inside. It was a division, the size of Android and docs and. A really sizable thing. They're under their own buildings.”
Gilad: 85% to 88% of users love Gmail's tabbed inbox
“Like splitting promotions and social to the side sounds like the stupidest idea, but there's about 85% of the population, 85 to 88% that absolutely love it. And today Gmail has about 1.8 billion active users, according to Gmail. Most of these users are using t…”
Gilad: Opinion-based debates mean the most senior person always wins
“Because when you have the discussion, you come with your own opinions. Usually the most senior person's opinions will win. That's just the way it is.”
Gilad: Measuring engineers solely on output drives disengagement
“So the engineers are kind of getting the signal that what they need to do is deliver. They're focused on output. That's what they're measured on. So they're kind of disengaged. They're disengaged from the users, from the business. They don't care that much.”
Gilad: Most Strategy Is Merely Execution Planning on Steroids
“In many companies, the strategy is just
A roadmap on steroids. It's more plan and execute just on a grand scale.”
Gilad: WhatsApp long measured messages sent as its primary user-value metric
“For example, let's take WhatsApp WhatsApp for very long time measured messages sent because every message sent is a little increment of value for the sender.”
Gilad: Amplitude Tracks 'Weekly Active Learning Users' Who Share Insights
“In amplitude, for example they measure learn active learning users or weekly active learning users, which are users that found in the tool, some insight that was so important that they shared it with at least two other users and they consume it.”
Gilad: Teams should structure topology around metrics trees instead of software architecture
“Another thing that happens in a lot of organization, the team topology reflects, you know, the structure of the software or some hierarchical model where we want to organize the organization in a particular way. But if you start with a metrics tree, you can tr…”
Gilad: Teams subvert ICE scoring by using gut feel for confidence
“What happens is that people tend to just go with gut instinct and then give themselves a high confidence. They say it's an eight and I'm pretty convinced. So it's eight for confidence. And I found this a bit disturbing cause it's kind of subverts the whole sys…”
Gilad: Validating features by copying competitors never works
“In many companies I meet, if the leading competitor has this feature and we think it's a good idea, validation is done. Let's launch it. That's it. It's a great idea. We need to do it. Never works, honestly. You should not assume that your competitor actually …”
Gilad: Heavy OKRs and full metric trees are overkill for early-stage startups
“Building your entire metrics trees, maybe overkill doing heavy weighted OKRs, maybe overkill for early stage companies even don't know how they create value. So they need to iterate and their goals is really to find product market fit”
Gilad: Most organizations treat MVPs as rebranded beta releases
“One of the patterns I see is that organizations don't know that they can actually learn at a much lower cost. They believe they need to build this elaborate MVP, which is not minimal in any way. And then launch it, and then they will discover it. Basically, it…”
Gilad: Gmail's tabbed inbox was first tested via an HTML facade
“We used a lot of those in the tabbed inbox, by the way, one of the first early versions was actually, we showed the tabbed inbox working. To people, but it wasn't really Gmail. It was just a facade of HTML and behind the scenes. And according to the permission…”
Gilad: Product teams cannot deliver on more than four key results
“These are the maximum four key results and the one or two objectives you guys need to work on teams cannot deliver on more than that.”
Gilad: Date-based release roadmaps eliminate learning and evidence-guided product development
“I would say that release roadmaps where you are just saying by Q three, we want to launch this or by October, we have to launch that. They're kind of competing with this. If you're doing that. And people know that the goal is to launch that thing by October. F…”
Gilad: Niche product design prompts effectively test PM empathy and creativity
“I like to ask them to design something for a niche audience. So a navigation system for elderly people or some sort of laptop for people with a vision impairment, et cetera. So those are good questions to see their customer empathy, their creativity, their abi…”