Disclosure
Shopify Has Banned Traditional KPIs and OKRs
“We rant against KPIs are basically banned as a, and OKRs are banned and all that.”
Insight
Long-Term Growth Experiment Results Frequently Diverge From Initial Wins
“Look at some of the experiments that you thought were your biggest winners. Look at the downstream metrics for a year, two years on that experiment, and I'll bet you'd be surprised how many times the metric is different than what you thought it would be after …”
Insight
Reducing Upfront Monetary Friction Causally Improves Merchant Success
“If you think about in a business case, if I give you A little less, a little monetary boost and reduce that monetary friction. I can actually causally change your ability to become successful. Cause I've given you a little bit more time to try that idea a litt…”
Assertion Not checkable as stated
30% to 40% of Short-Term Experiment Wins Have Zero Long-Term Impact
“Yeah, it's right. It's in the 30 to 40% range.”
Insight
Growth Teams Should Optimize for Absolute Volume Over Conversion Rates
“Orienting every team to think about the total number of people, not the rate, but the total number of people Who got to the end of their kind of part of the journey is a very powerful way to incentivize people to do the right thing in terms of getting people s…”
Insight
Reducing Top-of-Funnel Friction Drops CAC Faster Than It Hurts LTV
“And a lot of teams get very nervous. Their retention rate went down. Their LTV went down. Oh my goodness. This is going to affect our ability to pay. No, your CAC also went down by probably more. And so now you have the ability to likely spend more. And you ha…”
Insight
Technical Architecture Determines Tech Strategy More Than Target Users
“The how the technical architecture determines strategy in a technology company even more than the kind of what and who we're building for. If you build the right kind of technical how and set yourself up to have a platform that can be adaptable, flexible, that…”
Disclosure
Shopify Enforces a Strict 'No Wizard' Onboarding Product Principle
“We have very specific no wizard principle, but I think that sometimes the tension is Wizards do serve a, can serve a purpose in certain circumstances, and so we've, but we've avoided doing that, but we've always worked to try to make the principles of what a w…”
Insight
Purchasing Education Products Satisfies an Emotional Job Even Without Completion
“Education is very aspirational. And so what a lot of people missed in that, in education is, yes, we want people to actually take the course, but that's actually, in many cases, not the job to be done, that there's an emotional job that's even more important, …”
Insight
Power-Law Merchant GMV Drives Shopify Cohort Returns
“GMV is a power law based metric, and so it's really that power law that drives the success of each cohort.”
Insight
SaaS Products Should Collect More Onboarding Info to Personalize Guidance
“Collecting the right information, and you usually want to collect more information than most people think you do in your signup flow. If you can then leverage that to personalize the guidance, and this is for a SaaS product, the guidance that someone can get w…”
Insight
Shopify Ships Neutral A/B Tests if Product Intuition is Positive
“If the intuition is right, that this probably is helping merchants, Why do we start with that? The original control is better if it's neutral. Let's start with like, what would we have shipped if we were a blank slate? And if it's neutral, actually neither is …”
Insight
Metric-Free Product Development Requires Strict Central Gatekeepers of Taste
“I think if it's just subjective, let's just ship what we want without kind of a couple of people really holding that, that quality and that taste bar, that's where things go really sideways.”
Insight
Shopify Skips a CMO Because Its Founders Drive the Brand Story
“I think it only works because Toby, Harley have such amazing Intuition on what the brand is, needs to be, and all of that, that some of what the CMO does of kind of creating the cohesive story of Shopify, It's kind of held in, in their heads and they kind of, …”
Insight
Ship Upper-Funnel Winners Without Overestimating Long-Term Impact
“If something is showing lift up funnel, still ship it. And it's probably not going to hurt you, but don't overestimate the amount of impact that this is having.”
Disclosure
Shopify Uses Two Holdout Layers to Measure Long-Term Experiment Effects
“We have two layers of holdouts. So one is the, like more of the holdouts of like every change in a quarter hold out five percent across the board. Second is for changes that only affect new merchants, what we'll do is We'll take that group of folks, let's call…”
Disclosure
Shopify Pings Experimenters With Cohort Results Up to a Year Later
“In our experimentation tools, all experimenters are pinged at three months, six months, nine months, 12 months with your, the updated results. So you can't really get hide from What did this really result in over a longer term horizon?”
Insight
Constricting the Top of the Funnel Artificially Inflates Downstream Retention
“The easiest way to increase retention is always to constrict the funnel stage one above the retention metric you're trying to optimize for. The simplest way to increase my sign up to activated thing is just make it harder to sign up.”
Assertion Not checkable as stated
Payment Failure Alerts Gave Shopify Zero Long-Term GMV Lift
“We were doing much better alerting, reminding people, sending them a million emails about it. Cool. We got some pretty major lift. You look back six, 12 months. There was really no long-term left.”
Assertion Not checkable as stated
Pre-Configured Store Themes Drove Long-Term GMV Despite Neutral Initial Conversion
“We tested, okay, if we give you a pre-configured block of like, you should have an image up top, then a text banner, and then a collage with your products that should help folks understand what to do when they're building the online store. It actually had no l…”
Disclosure
Every Shopify Project Requires Explicit Group Lead Approval to Ship
“Every single project that goes out has a few minute video with figmas and everything and everything that shipped needs to be okay to, so approved by the group. There's nothing that can ship without that okay to approval. And that okay to approval has to be Gle…”
Assertion Not checkable as stated
Shopify Embeds Marketing Across Business Units Instead of Using a CMO
“There's growth marketing who sits in, in growth. There's revenue marketing who sits over closer to sales. There's a brand team under Harley. He does amazing work, our president. There's marketing embedded in core and PMM sit with product managers there. There'…”
Insight
Discounting Uses High List Prices to Signal Quality While Enabling Conversion
“What discounting has a really powerful effect on is it can signal value with a high list price, but then bring something down to an affordable price.”
Insight
Modern Marketing Has Lost First Principles to Algorithmic Optimization
“I find it really fun to go back in time because there's a lot of really good first principles thinking that I think we've actually lost in more modern stuff where it's like personalization bandits, optimization, all this stuff, or it's like, no, like what, how…”