Shopify Pings Experimenters With Cohort Results Up to a Year Later
Archie Abrams · Breaking the rules of growth: Why Shopify bans KPIs, optimizes for churn, and prioritizes intuition · Nov 7, 2024 · at 21:55
Archie Abrams, VP of Product and Head of Growth at Shopify, describes how Shopify holds product teams accountable for long-term experiment results.
0:00 / 0:15exact quote · 15.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“In our experimentation tools, all experimenters are pinged at three months, six months, nine months, 12 months with your, the updated results. So you can't really get hide from What did this really result in over a longer term horizon?”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Archie Abrams
Shopify Has Banned Traditional KPIs and OKRs
“We rant against KPIs are basically banned as a, and OKRs are banned and all that.”
Long-Term Growth Experiment Results Frequently Diverge From Initial Wins
“Look at some of the experiments that you thought were your biggest winners. Look at the downstream metrics for a year, two years on that experiment, and I'll bet you'd be surprised how many times the metric is different than what you thought it would be after …”
Reducing Upfront Monetary Friction Causally Improves Merchant Success
“If you think about in a business case, if I give you A little less, a little monetary boost and reduce that monetary friction. I can actually causally change your ability to become successful. Cause I've given you a little bit more time to try that idea a litt…”
30% to 40% of Short-Term Experiment Wins Have Zero Long-Term Impact
“Yeah, it's right. It's in the 30 to 40% range.”
Growth Teams Should Optimize for Absolute Volume Over Conversion Rates
“Orienting every team to think about the total number of people, not the rate, but the total number of people Who got to the end of their kind of part of the journey is a very powerful way to incentivize people to do the right thing in terms of getting people s…”
Reducing Top-of-Funnel Friction Drops CAC Faster Than It Hurts LTV
“And a lot of teams get very nervous. Their retention rate went down. Their LTV went down. Oh my goodness. This is going to affect our ability to pay. No, your CAC also went down by probably more. And so now you have the ability to likely spend more. And you ha…”