Schaffer: Microsoft and Netflix experimentation shows 80% to 90% of hypotheses fail
Laura Schaffer · Career frameworks, A/B testing, onboarding tips, selling to engineers | Laura Schaffer (Amplitude) · Mar 9, 2023 · at 34:52
Amplitude Head of Growth Laura Schaffer explains why iterative experimentation is essential by citing published failure rates from major tech firms.
0:00 / 0:07exact quote · 7.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Companies like Netflix and Microsoft, there's over and over again. 80 plus percent. Some companies say 90% of things fail.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Laura Schaffer
Schaffer: Growth teams should accept lower confidence intervals to run more experiments
“And so there are definitely times where I will advocate for and I will push for and I will myself use. Lower, lower confidence intervals than 95%, especially if that, like, doubles the amount of experiments that you can run in a year, right? End of the day, th…”
Schaffer: Adding onboarding questions boosted Twilio signups by five percent
“I'm not kidding. It improved conversion. Like there's no personalization, nothing past it, just the questions. It improved conversion by like five percent, like just improved signups.”
Schaffer: Evaluating Growth Teams on Short Timelines Encourages Vanity Metrics
“Any growth team that's beholden to short timeline wins and improvement is always going to be dangerous. That's a, that's an environment that's conducive to vanity metric usage and fabric, you know, massaging the data to fit.”
Schaffer: Moving to PLG requires redefining customer problems, not unbundling enterprise products
“The thing that I think sticks across all companies, not just Amplitude making the shift is just that, that when you're doing this, do not think this is a copy paste, but like chop it for parts thing. Don't start with your product when you're building out your …”
Schaffer: Executive disconnect from customers creates career leverage for frontline employees
“Your executive team and executive teams at companies are often very sharp, but the nature of their day to day just does not link them with customers. Right. And that means that over time, especially as a company grows, they often lose access to some of the bes…”
Schaffer: Market expansion and product additions increase developer onboarding friction
“As we were penetrating new markets, adding more products, we were adding complexity and we were pulling in folks who were a little bit less motivated. And those things contributed to people saying, this is difficult.”