Sep 12, 2018 · 59m · y-combinator
Suhail Doshi - How to Measure Your Product · Y Combinator
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this Y Combinator Startup School lecture, Mixpanel co-founder Suhail Doshi presents a practical framework for measuring product success centered on a five-step growth funnel, core retention metrics, and single North Star alignment. He provides actionable guidance on eliminating onboarding friction, leveraging qualitative user feedback at early stages, and avoiding overcomplicated metrics.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The partners as informed peer 0.1 Guest teaching 0.0 Guest disagreement 0.5 The partners pushing back 0.0
speaking balance: gold is the partners, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 27:20 Calling MAU a BS vanity metric
Suhail bluntly rejects conventional monthly active user reporting, calling it a meaningless vanity metric akin to total registered accounts.
Hardest push from the partners ▶ 45:50 Audience question on churn mathematics discrepancyAn audience member challenges Suhail on how 7 percent monthly churn translates to 58 percent revenue loss when retention benchmarks cite 35 percent as healthy.
Biggest teaching moment ▶ 27:55 Compounding monthly churn mathematics lessonSuhail educates the audience on why 7 percent monthly churn is lethal, demonstrating that it quietly wipes out nearly 60 percent of annual revenue.
The partners hold their own ▶ 0:00 Host introductory framing on product measurement and growthThe host frames product measurement as the essential foundational counterpart to startup growth and building what users truly want.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The partners as informed peer | Guest teaching | Guest disagreement | The partners pushing back | Why |
|---|---|---|---|---|---|---|
| Title Slide and Audience User Survey | 0 | 0 | 0 | 0 | Suhail delivers a solo presentation opening, polling the audience on their user counts and sharing Mixpanel background context. The host does not participate after the introductory remark. | |
| The Top-Down Framework and Growth Formula | 0 | 0 | 0 | 0 | Suhail outlines his top-down five-part growth formula from first principles. This is a structured lecture monologue without host interaction. | |
| Avoiding Overcomplicated Metrics | 0 | 0 | 1 | 0 | Suhail critiques companies that track 25 disparate submetrics, pointing out that excessive data tracking leads to organizational paralysis. The dynamic is purely didactic. | |
| Linking User Funnels to Core Questions | 0 | 0 | 0 | 0 | Suhail reviews the historical evolution of Mixpanel's landing pages and taglines to illustrate communicating value propositions clearly. | |
| How to Measure Product Understanding | 0 | 0 | 0 | 0 | Suhail shares practical metrics for testing whether visitors understand a product, including pricing page click-throughs in B2B and copy A/B testing. | |
| Streamlining Initial Onboarding and Getting Started | 0 | 0 | 1 | 0 | Suhail emphasizes ruthless prioritization during early onboarding, referencing how early Mixpanel skipped forgot password flows entirely. | |
| Measuring Onboarding Success and Conversion Tips | 0 | 0 | 1 | 0 | Suhail advises startups against premature fraud prevention like email confirmations that hurt conversion funnels. He speaks as a solo lecturer. | |
| User Retention and 'The Shark Fin Effect' | 0 | 0 | 1 | 0 | Suhail illustrates the shark fin effect where viral user growth collapses due to nonexistent retention, using LivingSocial as an example. | |
| Measuring Retention and Monthly Dollar Churn | 0 | 0 | 2 | 0 | Suhail dismisses MAU as a vanity BS metric, advocating instead for harsh DAU tracking and highlighting how a 7 percent monthly churn compounds to a 58 percent annual loss. | |
| Qualitative Analytics for Early-Stage Startups | 0 | 0 | 0 | 0 | Suhail advises sub-50 user startups to gather qualitative feedback over direct messaging rather than attempting statistical cohort analysis. | |
| Choosing a North Star Metric & Recommended Reading | 0 | 0 | 0 | 0 | Suhail concludes his prepared remarks by advising teams to pick a single company-defining North Star metric for six-month commitments and citing Andrew Chen's essay. | |
| Interactive Audience Q&A Session | 1 | 0 | 1 | 0 | Audience members ask practical questions regarding benchmarks, video onboarding, churn differentiation, and startup misconceptions. Suhail answers collegially, while the host solely acts as a mic moderator. | |
| End Screen and Subscription Prompt | 0 | 0 | 0 | 0 | Standard wrap up and session conclusion. No substantive dynamic or pushback. |
Statements from this episode (22)
Mixpanel nears $100M ARR with 7,000 paying customers
“10 years later, the company has about 300 employees. We've raised a bunch of money and we have about 7000 paying customers, and we're starting to close in on around a hundred million dollars in annual recurring revenue.”
Doshi: Mixpanel struggled for 18 months before users understood the product
“And Mixpell had a really hard time with that in its first 18 months. We had to change a lot of things and most people do.”
Doshi: Thousand-person companies regularly overcomplicate metric tracking
“Even thousand-person companies get the basics wrong. In fact, you'll find that if you don't get the, even these thousand-person companies that get it wrong, they tend to overcomplicate the number of things that they should actually measure and track.”
Doshi: Tracking three to five North Star metrics beats complex dashboards
“And so, what I find is that just picking Like three to five things on North Star and really simplifying it will do more good for you.”
Doshi: Startups lose most visitors because the back button is the enemy
“And I also wanted to mention that having a really bad landing page there's this really awesome quote that PG used to have in our batch that I thought I would impart on you, which is that, like, people basically just have their mouse, like, hovering on the back…”
Mixpanel differentiated from Google Analytics by measuring actions over page views
“Was that, ah, we transformed from metrics that'll make you drool to actions speak louder than page views. And you'll notice it's, like, extremely prominent. The reason why was because the number one question that people, that we would get as a company was, tha…”
Doshi: Pricing page links are the second or third most clicked in B2B
“One thing that I discovered was that people, a lot of your users will click on a pricing page link at the top. It's like second or third, like most clicked on link if you're a B to B company, I think, generally speaking.”
Doshi: B2B startups struggle to run A/B tests compared to consumer apps
“If you're in consumer the benefit of being in consumer is that you generally have you usually will have like hundreds or thousands of users such that you could actually go do an A-B test, whereas in B to B it's a lot harder to achieve that.”
Zenter (later Google Slides) launched without a save button to test demand
“I remember Robbie Walker during one of our one of one of our Tuesday dinners said that they had, they didn't even implement the save button for, like, their online PowerPoint because they knew that none of their users would ever ask, like, hey, can I save this…”
Mixpanel operated without a forgot password feature for 18 months
“For like, I don't know, maybe like, 18 months of Mixpanel, like, we didn't even have forgot about password, because we were just so ruthlessly trying to prioritize, like, things that really, would really matter to customers. And it turned out we didn't get ver…”
Doshi: Early startups should not optimize onboarding against fraud or spam
“You don't have lots of fraud problems. You don't have, you probably don't have a lot of people spamming your service. You probably don't have a lot of, like, fake actors in the system trying to sign up. So if you don't have those problems, don't optimize for t…”
Doshi: Email and text verifications lose 60% to 70% of onboarding users
“Generally email and text confirmations have an enormous drop-off. Like, I'm talking, like, you get, of the hundred percent of the people that come, like, only 30% will, like, basically go on and, like, click on those email confirmations, or 40%.”
Doshi: Products should allow user access before demanding sign-up
“To the extent that you can just let users into the product it's always better to do that.”
Doshi: Reactivation campaigns generally do not work
“If you talk to people that have ever done, like, a reactivation campaign, reactivation campaigns generally don't go very well.”
Doshi: LivingSocial pivoted to daily deals due to poor retention
“LivingSocial literally had to pivot the entire company to basically competing with Groupon because the app wasn't very retentive in those days.”
Doshi: Monthly active users has become a deceptive vanity metric
“I now think that monthly active users has become, like, the new BS metric that's, like, very similar to, like, just, like, number of registered users.”
Doshi: Having ~1.4 DAUs per account reduced Mixpanel's dollar churn below 10%
“We actually found that there was, like, a really, really strong correlation with daily use,
And rate of churn, like, rate of, like, people, like, stop not no longer paying for the product, and it didn't take very many users.
I think it was, like, one point, …”
A 7% monthly churn rate compounds to a 58% annual revenue loss
“If you have revenue churn, and your revenue churn is, say, seven percent per month, right, you might think, awesome, I'm keeping 93% of these dollars every single month.
This is awesome.
I'm doing really, really, really well.
The problem is, is it's seven p…”
Doshi: Talking to users yields far more insight than analyzing data graphs
“So, I don't think there's any way that you can kind of skirt by talking to customers, but here's the benefit of talking to customers, even though it feel, maybe it's more tedious than looking at a graph you're going to get way more information and depth from t…”
Doshi: Startups should commit to one North Star metric in six-month blocks
“Pick one North Star metric, and I think, and a North Star metric I would choose in this case is like, what is a number that you're willing to bet the company on? Right? Like, if that number goes south, you deserve to die, and if that number goes up, you will, …”
Mixpanel and Slack both converted 4-5% of landing page visits to signups
“Going from like a visit to a signup, like I think like a four to five percent conversion rate is pretty good. I think I like swap notes with Stuart at Slack cause we kind of had similar landing pages. I think I asked him, you know, what was your conversion rat…”
Doshi: Founders often fail by getting distracted instead of focusing for years
“I think it's really hard for the average founder to actually just be focused on something boring. We tend to have, like, ADD or something, and, like, we just decide, decide to get very easily distracted and I think it's hard to do something, do the same thing …”