Mar 9, 2023 · 1h 21m · lennys-podcast
Career frameworks, A/B testing, onboarding tips, selling to engineers | Laura Schaffer (Amplitude)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of Lenny's Podcast, Amplitude Head of Growth Laura Schaffer shares tactical frameworks for career acceleration through customer insights, the counterintuitive power of 'good friction' in user onboarding, and high-velocity experimentation strategies for product-led growth.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 18.6% of the talking time here. How this is scored →
Lenny as informed peer 4.0 Guest teaching 5.5 Guest disagreement 0.9 Lenny pushing back 0.6
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 44:20 Laura disputes traditional statistical rigor standards
Laura directly challenges industry-standard 95% confidence intervals, arguing they hinder experiment velocity and are unnecessary outside medical contexts.
Hardest push from Lenny ▶ 44:42 Lenny highlights data science resistance to lower p-valuesLenny pushes on the controversial nature of lowering confidence intervals, noting data science teams would typically push back hard against the approach.
Biggest teaching moment ▶ 23:15 Laura educates on the psychological benefit of signup frictionLaura demonstrates that adding onboarding friction increased signups by 5% because answering tailored questions reassured anxious users rather than burdening them.
Lenny holds their own ▶ 36:22 Lenny cites Airbnb redesign failures to validate iterationLenny draws on his past Airbnb growth experience to validate Laura's thesis, asserting that massive product redesigns fail essentially 100% of the time.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| Welcoming Laura Schaffer to Lenny's Podcast | 4 | 5 | 1 | 1 | Lenny welcomes Laura and sets up the topic of career growth. Laura explains her framework of proactively surfacing customer insights rather than passively relying on manager promotion cycles. | |
| Founding Twilio's Growth Team with Customer Voice Reports | 3 | 6 | 1 | 0 | Laura shares the case study of starting Twilio's growth team by producing customer feedback digests that challenged internal executive beliefs about product ease of use. Lenny listens and prompts for details. | |
| Ungating Knowledge and Accelerating Internal Career Growth | 4 | 4 | 0 | 0 | Lenny summarizes Laura's advice as taking initiative, and Laura adds the principle of 'ungating your knowledge' across the company. | |
| Good Friction vs. Bad Friction in Signup Onboarding | 4 | 7 | 2 | 1 | Laura details an experiment where adding questions into signup onboarding surprisingly increased conversion by 5%, debunking the conventional dogma that all signup friction is bad. | |
| The 'Pill in the Hot Dog' Onboarding Technique | 4 | 6 | 1 | 0 | Laura details the 'pill in the hot dog' metaphor, explaining how burying intimidating telecom steps inside familiar coding documentation boosted activation. Lenny adds his household anecdote about using peanut butter. | |
| Embracing High Failure Rates and Iterative Testing Over Redesigns | 6 | 5 | 1 | 2 | Lenny shares his Airbnb experience that full redesigns fail 100% of the time, reinforcing Laura's point that teams should expect an 80% baseline experiment failure rate and iterate quickly. | |
| Rapid Validation: Pre-Testing Hypotheses Before A/B Tests | 4 | 5 | 1 | 1 | Laura argues that because A/B testing is expensive, teams should pre-validate hypotheses using painted doors and design mocks before writing code. | |
| Sponsor Break: Writer Enterprise Generative AI Platform | 5 | 7 | 2 | 2 | After an ad break, Laura introduces the controversial idea of relaxing the 95% statistical confidence threshold in product growth to double testing velocity. Lenny highlights the boldness of the stance. | |
| Setting Annual Growth Horizons Over Short-Term Metric Pressure | 4 | 6 | 1 | 0 | Laura warns against evaluating growth teams on weekly sprints, explaining that high failure rates necessitate annual horizons to avoid vanity metric manipulation. | |
| Case Study: Twilio Code Exchange and Low-Code Activation | 4 | 6 | 0 | 0 | Laura recounts building Twilio Code Exchange Quick Deploy, a low-code onboarding flow that drove tens of millions in enterprise pipeline by empowering non-technical evaluators. | |
| Navigating the Shift from Sales-Led to Product-Led Growth | 5 | 6 | 1 | 1 | Laura critiques enterprise companies that try to transition to PLG by merely chopping up their existing product, emphasizing the necessity of understanding the distinct psychology and resource constraints of self-serve users. | |
| Understanding Developer Psychology When Selling Technical Products | 4 | 7 | 2 | 0 | Laura breaks down developer psychology, illustrating why engineers bypass marketing websites and sales reps entirely due to the extreme personal and operational accountability they carry. | |
| Lightning Round: Top Books, Culture, and Software Stack | 3 | 3 | 0 | 0 | Lenny runs through the lightning round questions covering books, reality TV psychology, interview questions, and recommended software tooling. | |
| Amplitude Power Tip: Combining Funnel Metrics with Session Replays | 2 | 4 | 0 | 0 | Laura shares an Amplitude pro tip on chaining quantitative funnel anomalies directly to qualitative session replays via Segment and Hotjar before wrapping up the episode. |
Statements from this episode (17)
Schaffer: Executive disconnect from customers creates career leverage for frontline employees
“Your executive team and executive teams at companies are often very sharp, but the nature of their day to day just does not link them with customers. Right. And that means that over time, especially as a company grows, they often lose access to some of the bes…”
Schaffer: Market expansion and product additions increase developer onboarding friction
“As we were penetrating new markets, adding more products, we were adding complexity and we were pulling in folks who were a little bit less motivated. And those things contributed to people saying, this is difficult.”
Schaffer: Customer-facing staff hold strong leverage to break into product
“Customer facing teams, such as those who want to crack into product. Your insights are extremely valuable. You're talking to customers every day. You know more about their problems and their pain than a lot of other people do.”
Schaffer: Peer leadership exposure makes promotion calibrations much easier for managers
“One of the things that, you know, It comes into play a lot when managers are figuring out promotions and doing those things. It's like they'll sit in a room off on calibrations and, you know, with a bunch of people, and it makes it a lot easier when those peop…”
Schaffer: Adding onboarding questions boosted Twilio signups by five percent
“I'm not kidding. It improved conversion. Like there's no personalization, nothing past it, just the questions. It improved conversion by like five percent, like just improved signups.”
Schaffer: Poor conversion in logical funnels signals psychological friction
“And if something like very logical isn't converting well, sometimes it means that you're battling against the psyche of the user and you want to take a step back and You know, think about and learn about where someone is psychologically in your space.”
Schaffer: Microsoft and Netflix experimentation shows 80% to 90% of hypotheses fail
“Companies like Netflix and Microsoft, there's over and over again. 80 plus percent. Some companies say 90% of things fail.”
Rachitsky: Complete product redesigns yield negative results 100% of the time
“In my experience, launching a whole redesign is, is negative. A hundred percent of the time I've been, I've grown weary to avoid that as much as possible”
Schaffer: A/B testing is one of the most expensive validation methods
“A-B testing is one of the most expensive kinds of ways to validate an experiment, right? It you know, often requires design and engineering and the PM or growth person or marketing person who's crafting it, right? All these things Or investments that take a lo…”
Schaffer: Growth teams should accept lower confidence intervals to run more experiments
“And so there are definitely times where I will advocate for and I will push for and I will myself use. Lower, lower confidence intervals than 95%, especially if that, like, doubles the amount of experiments that you can run in a year, right? End of the day, th…”
Schaffer: Evaluating Growth Teams on Short Timelines Encourages Vanity Metrics
“Any growth team that's beholden to short timeline wins and improvement is always going to be dangerous. That's a, that's an environment that's conducive to vanity metric usage and fabric, you know, massaging the data to fit.”
Schaffer: Growth teams should set annual low-medium-high target ranges
“I think it's good, especially for newer teams, but even teams in general commit, commit to something that you can kind of do over the course of a year and, you know, low, medium, high, right.”
Schaffer: Moving to PLG requires redefining customer problems, not unbundling enterprise products
“The thing that I think sticks across all companies, not just Amplitude making the shift is just that, that when you're doing this, do not think this is a copy paste, but like chop it for parts thing. Don't start with your product when you're building out your …”
Schaffer: Developers bypass marketing websites to go straight to product signup
“Developers, like almost two one, do not look at your marketing website at all. They go straight to your signup flow. So what that means is all that beautiful context that you're setting and, you know, the product and pricing and all that stuff, like very, you …”
Schaffer: A FAANG company deployed Twilio without ever speaking to sales
“There was a Fang company who signed up for Twilio. Built a POC, launched to production, all this, and operate in that space for months without engaging once with sales.”
Schaffer: Ask candidates about recent releases rather than cherry-picked successes
“The thing is everyone has like a big success story. Everyone does. So it really doesn't actually tell you very much to ask someone like, what's a great thing you released? Cause everyone can tell that. Instead, take, take that away. What's most recent ship is …”
Schaffer: Integrating analytics with session replays explains user anomalies without interviews
“You know, talking to customers is very powerful, but in this case, just that simple use of kind of connecting and threading those technologies together could really get a good picture of that without needing to engage customers.”