Nov 7, 2024 · 1h 17m · lennys-podcast
Breaking the rules of growth: Why Shopify bans KPIs, optimizes for churn, and prioritizes intuition
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Archie Abrams, VP of Product and Head of Growth at Shopify, reveals the unconventional growth philosophies behind the e-commerce giant's immense scale. He details why Shopify intentionally embraces merchant churn, bans KPIs in core product teams, measures experiments using multi-year holdouts, and prioritizes absolute volume and technical craft over local conversion rates.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 34.1% of the talking time here. How this is scored →
Lenny as informed peer 4.8 Guest teaching 6.2 Guest disagreement 1.7 Lenny pushing back 1.2
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:05:25 Archie's critique of multi-touch attribution
Archie rejects conventional multi-touch attribution models, arguing forcefully that only causal incrementality testing provides true measurement.
Hardest push from Lenny ▶ 11:09 Lenny challenges multi-year feedback loopsLenny pushes back on Archie's long-term cohort model, pointing out that five-year feedback loops are seemingly unusable for operational product decisions.
Biggest teaching moment ▶ 17:49 The 30-40% decay of growth experiment winsArchie educates Lenny and the audience by revealing that up to 40% of short-term experiment wins show zero incremental GMV when tracked over a multi-year holdout.
Lenny holds their own ▶ 46:54 Lenny contrasts Tobi's architecture strategy with Brian Chesky's design-led modeLenny demonstrates deep product expertise by synthesizing Archie's points about Tobi Lütke and comparing them directly to Brian Chesky's design-oriented founder mode at Airbnb.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| Welcome and the Staggering Scale of Shopify | 3 | 4 | 1 | 1 | Lenny opens the interview by setting up the theme of Shopify's counterintuitive scale and growth practices. Archie shares staggering stats on Shopify handling 10% of US e-commerce and $235B in GMV. | |
| Optimizing for Churn and Power Law Monetization | 5 | 6 | 2 | 2 | Lenny asks how Shopify can optimize for churn rather than retention. Archie explains that lowering barriers allows a power-law distribution where breakout merchants make the entire cohort profitable through payments monetization. | |
| The 100-Year Horizon and Multi-Year Holdout Experiments | 5 | 7 | 2 | 3 | Lenny pushes back on how actionable a multi-year feedback loop can be. Archie explains how Shopify splits horizons between core, merchant services, and growth, using automated long-term holdouts to hold teams accountable. | |
| Downstream Decay of Experiment Wins and Monetary Friction | 5 | 8 | 3 | 2 | Archie drops the sobering fact that 30 to 40 percent of experiments showing initial positive lift have zero long-term impact after a year. Lenny notes this will break the hearts of most growth professionals. | |
| Navigating Funnels and Deciding What to Ship | 4 | 5 | 1 | 1 | Lenny asks practical guidance for companies unable to wait years for holdouts. Archie advises shipping short-term winners while remaining sober about their true downstream magnitude. | |
| Shopify's Internal Experimentation Infrastructure | 4 | 6 | 1 | 1 | Archie describes Shopify's custom experimentation tooling, including automated alerts sent to teams at 3, 6, 9, and 12 months with cohort performance updates. | |
| High-Impact Levers: Personalized Onboarding vs Funnel Traps | 5 | 7 | 3 | 1 | Archie details high-impact onboarding wins and warns against the trap of optimizing local conversion rates by constricting upper-funnel volume. | |
| Focusing on Absolute Volume Over Conversion Percentages | 5 | 6 | 2 | 1 | Archie explains monetary friction levers and why growth teams should measure absolute incremental merchants rather than percentage conversion ratios. | |
| Shopify's Growth Org Structure and Cohort Value Metrics | 5 | 6 | 1 | 1 | Archie explains the 600-person growth team structure spanning Growth R&D (product, enable, support tooling) and Growth Marketing. Lenny synthesizes the organizational pillars. | |
| Sponsor Segment: Dovetail Customer Insights | 4 | 7 | 2 | 1 | Following the Dovetail sponsor spot, Archie illustrates downstream decay with a failed payment alerting experiment that had zero 12-month impact, contrasted with editor templates that initially showed neutral lift but massive long-term GMV gains. | |
| Tobi Lütke's 100-Year Philosophy and Technical Architecture as Strategy | 6 | 7 | 1 | 1 | Archie details Tobi Lütke's belief that technical architecture dictates strategy. Lenny draws on his own Airbnb experience to contrast Tobi's engineering-first philosophy with Brian Chesky's design-led founder mode. | |
| Banning KPIs in Core Product and the GSD Review Process | 6 | 7 | 2 | 1 | Archie explains why OKRs and KPIs are banned in core product, relying instead on high-taste leadership and the six-week GSD review system. Lenny connects this to delegated founder mode. | |
| Balancing Growth and Core: The "No Wizard" Principle | 5 | 6 | 2 | 1 | Archie describes managing the deliberate tension between core product and growth, highlighting the 'no wizard' rule and embedding onboarding defaults directly into the main UI. | |
| Taste vs Metrics: When Non-KPI Product Building Works | 6 | 6 | 2 | 1 | Lenny and Archie debate when non-KPI product building works, concluding it requires either rare high-taste founders or rigorous metric accountability to prevent haphazard execution. | |
| Expanding into Enterprise: Sales, Hybrid Journeys, and Incrementality | 6 | 7 | 3 | 1 | Archie discusses enterprise expansion, hybrid self-serve/sales funnels, and passionately argues why causal incrementality testing beats traditional multi-touch attribution. | |
| Why Shopify Operates Without a Chief Marketing Officer | 5 | 6 | 1 | 1 | Archie explains why Shopify operates without a CMO by embedding marketing disciplines into dedicated product and business units under executive brand stewardship. | |
| The Psychology of Discounting: Insights from Udemy | 4 | 7 | 1 | 1 | Archie shares lessons from Udemy on discounting psychology, explaining price anchoring and how purchasing a course fulfills an emotional job of perceived personal progress. | |
| Lightning Round: Book Recommendations, Media, and Life Lessons | 4 | 4 | 0 | 0 | Lenny and Archie close with the lightning round, covering vintage advertising books, marathon running literature, Sopranos, Suno AI, and leadership lessons on empathy from Archie's late father. |
Statements from this episode (29)
Shopify Powers 10% of US E-Commerce
“So overall we're about 10% of e-commerce in the United States.”
Shopify Generated $235B Global GMV in 2023
“Globally, we did about two hundred thirty five billion in GMV in 2023, which is roughly the size of the economy of Finland.”
Most Shopify Revenue Comes From Payments, Not Subscriptions
“And the way the Shopify business model works is we do charge a subscription, but most of our revenue comes from payments, which is tied to directly to a merchant's success.”
Power-Law Merchant GMV Drives Shopify Cohort Returns
“GMV is a power law based metric, and so it's really that power law that drives the success of each cohort.”
Shopify Divides Product Teams Across Three Time Horizons
“There's three major product groups at Shopify. There's core product, which is basically building the hundred year, the right things for commerce, a hundred years from now. There's merchant services, which is building things like payments, shipping, kind of the…”
Long-Term Growth Experiment Results Frequently Diverge From Initial Wins
“Look at some of the experiments that you thought were your biggest winners. Look at the downstream metrics for a year, two years on that experiment, and I'll bet you'd be surprised how many times the metric is different than what you thought it would be after …”
Reducing Upfront Monetary Friction Causally Improves Merchant Success
“If you think about in a business case, if I give you A little less, a little monetary boost and reduce that monetary friction. I can actually causally change your ability to become successful. Cause I've given you a little bit more time to try that idea a litt…”
30% to 40% of Short-Term Experiment Wins Have Zero Long-Term Impact
“Yeah, it's right. It's in the 30 to 40% range.”
Ship Upper-Funnel Winners Without Overestimating Long-Term Impact
“If something is showing lift up funnel, still ship it. And it's probably not going to hurt you, but don't overestimate the amount of impact that this is having.”
Shopify Uses Two Holdout Layers to Measure Long-Term Experiment Effects
“We have two layers of holdouts. So one is the, like more of the holdouts of like every change in a quarter hold out five percent across the board. Second is for changes that only affect new merchants, what we'll do is We'll take that group of folks, let's call…”
Shopify Pings Experimenters With Cohort Results Up to a Year Later
“In our experimentation tools, all experimenters are pinged at three months, six months, nine months, 12 months with your, the updated results. So you can't really get hide from What did this really result in over a longer term horizon?”
SaaS Products Should Collect More Onboarding Info to Personalize Guidance
“Collecting the right information, and you usually want to collect more information than most people think you do in your signup flow. If you can then leverage that to personalize the guidance, and this is for a SaaS product, the guidance that someone can get w…”
Constricting the Top of the Funnel Artificially Inflates Downstream Retention
“The easiest way to increase retention is always to constrict the funnel stage one above the retention metric you're trying to optimize for. The simplest way to increase my sign up to activated thing is just make it harder to sign up.”
Growth Teams Should Optimize for Absolute Volume Over Conversion Rates
“Orienting every team to think about the total number of people, not the rate, but the total number of people Who got to the end of their kind of part of the journey is a very powerful way to incentivize people to do the right thing in terms of getting people s…”
Reducing Top-of-Funnel Friction Drops CAC Faster Than It Hurts LTV
“And a lot of teams get very nervous. Their retention rate went down. Their LTV went down. Oh my goodness. This is going to affect our ability to pay. No, your CAC also went down by probably more. And so now you have the ability to likely spend more. And you ha…”
Payment Failure Alerts Gave Shopify Zero Long-Term GMV Lift
“We were doing much better alerting, reminding people, sending them a million emails about it. Cool. We got some pretty major lift. You look back six, 12 months. There was really no long-term left.”
Pre-Configured Store Themes Drove Long-Term GMV Despite Neutral Initial Conversion
“We tested, okay, if we give you a pre-configured block of like, you should have an image up top, then a text banner, and then a collage with your products that should help folks understand what to do when they're building the online store. It actually had no l…”
Shopify Ships Neutral A/B Tests if Product Intuition is Positive
“If the intuition is right, that this probably is helping merchants, Why do we start with that? The original control is better if it's neutral. Let's start with like, what would we have shipped if we were a blank slate? And if it's neutral, actually neither is …”
Shopify R&D Leads Review Every Project With Tobi Lütke Every Six Weeks
“Every six weeks, we, all the kind of R and D group leads, we get together and we sit with Toby and each other and review every single project across the company every six weeks. Every single R and D, pull up the dashboard and look at it.”
Technical Architecture Determines Tech Strategy More Than Target Users
“The how the technical architecture determines strategy in a technology company even more than the kind of what and who we're building for. If you build the right kind of technical how and set yourself up to have a platform that can be adaptable, flexible, that…”
Shopify Has Banned Traditional KPIs and OKRs
“We rant against KPIs are basically banned as a, and OKRs are banned and all that.”
Metric-Free Product Development Requires Strict Central Gatekeepers of Taste
“I think if it's just subjective, let's just ship what we want without kind of a couple of people really holding that, that quality and that taste bar, that's where things go really sideways.”
Every Shopify Project Requires Explicit Group Lead Approval to Ship
“Every single project that goes out has a few minute video with figmas and everything and everything that shipped needs to be okay to, so approved by the group. There's nothing that can ship without that okay to approval. And that okay to approval has to be Gle…”
Shopify Enforces a Strict 'No Wizard' Onboarding Product Principle
“We have very specific no wizard principle, but I think that sometimes the tension is Wizards do serve a, can serve a purpose in certain circumstances, and so we've, but we've avoided doing that, but we've always worked to try to make the principles of what a w…”
Shopify Embeds Marketing Across Business Units Instead of Using a CMO
“There's growth marketing who sits in, in growth. There's revenue marketing who sits over closer to sales. There's a brand team under Harley. He does amazing work, our president. There's marketing embedded in core and PMM sit with product managers there. There'…”
Shopify Skips a CMO Because Its Founders Drive the Brand Story
“I think it only works because Toby, Harley have such amazing Intuition on what the brand is, needs to be, and all of that, that some of what the CMO does of kind of creating the cohesive story of Shopify, It's kind of held in, in their heads and they kind of, …”
Discounting Uses High List Prices to Signal Quality While Enabling Conversion
“What discounting has a really powerful effect on is it can signal value with a high list price, but then bring something down to an affordable price.”
Purchasing Education Products Satisfies an Emotional Job Even Without Completion
“Education is very aspirational. And so what a lot of people missed in that, in education is, yes, we want people to actually take the course, but that's actually, in many cases, not the job to be done, that there's an emotional job that's even more important, …”
Modern Marketing Has Lost First Principles to Algorithmic Optimization
“I find it really fun to go back in time because there's a lot of really good first principles thinking that I think we've actually lost in more modern stuff where it's like personalization bandits, optimization, all this stuff, or it's like, no, like what, how…”