Isford: Prioritize Directional Signals Over Statistical Significance in Niche Experiments
Lauryn Isford · Lauryn Isford: Product Growth Secrets from Facebook, Airtable, BlueBottle, Dropbox & Notion | E1037 · 20VC with Harry Stebbings · Jul 19, 2023 · at 51:08
Lauryn Isford is the Head of Product Growth at Notion. She explains how growth teams should evaluate experiment duration and metric changes when testing on small user sets.
0:00 / 0:39exact quote · 39.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“In some cases, if you're making a change that's focused on a pretty narrow set of customers, you may never actually get to that significance. It might take months, right? Or even a year. So in general, I worry less about how long to run an experiment and think more about what are the key signals that will help us determine if the behavior we seek from our customers is happening or not happening. And then what are the guardrails that we want to look at? So even if you run an experiment for a week or two, you might see that more people are paying for your product or fewer people are paying for your product. And without significance, it could give you enough signal to say, let's keep moving and let's get this out the door and ship it to everyone.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Lauryn Isford
A heavy bias towards product experimentation is a growth hiring red flag
“The biggest red flag to me is when I hear a heavy bias towards experimentation. So maybe requiring every change made in the product to be an experiment or over focusing on optimizing, adding friction, removing friction raising prices, reducing prices versus ma…”
Tooltip pop-ups fail because users cannot process that much information quickly
“You know, you sign up for something and you see eight or 10 or maybe even 20 different pop-ups that define terms for you and ask you to learn a bunch of things, and the reality is that nobody's actually learning that many things that quickly.”
Isford: Consumer growth expertise does not translate directly to B2B SaaS
“One mistake I've seen is assuming that best growth practices and experience in consumer will translate to SaaS and the other way around. That risk was taken on me when I started at Airtable. I actually came from Meta, and so I was jumping from consumer to SaaS…”
SaaS growth requires bold product changes, not consumer-style micro-optimizations
“I find that in consumer growth, there tends to be an over-focus on very small detail changes. Whereas in SaaS, it's more important to make bigger or bolder changes that really react to the user's intent so that their behavior will change.”
PLG and sales-assisted growth motions inevitably compete for enterprise revenue
“At some point, the two motions will compete. So a dollar earned through a freemium or self-serve motion from PLG may be a trade-off against two or five dollars earned through closing a contract in a sales assisted motion. And it's very important to be explicit…”