StarZero All shows Try the podcast clipper →

Sep 28, 2017 · 20m · mad

How to Measure Product Success // Todd Olson, Pendo (FirstMark's Data Driven)

Matt Turck · 16m spoken
0:00 / 0:00
▶ Watch on YouTube →

gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions

Todd Olson, CEO of Pendo, presents a framework of five proxy metrics—Breadth, Frequency, Depth, Efficiency, and Satisfaction—to help product managers measure user engagement and product success before lagging indicators like revenue churn occur.

How this conversation actually went

Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 93% of the talking time here. How this is scored →

Matt as informed peer 0.5 Guest teaching 4.7 Guest disagreement 0.3 Matt pushing back 0.2
05100:0010:0020:000:27–3:32 · Matt as informed peer 0/10 Defining What Makes a Great Product Todd Olson presents on product quality, explaining that traditional metrics like Google Analytics miss the mark when measuring product experience. Host scores are 0 as this is a solo presentation.3:32–5:57 · Matt as informed peer 0/10 The 5 Key Measurements Framework Olson outlines the first two metrics, breadth and frequency, explaining why accounts with under two active users pose severe churn risks in B2B SaaS. Host scores remain 0 for monologue.5:57–8:56 · Matt as informed peer 0/10 Importance of Segment Analysis Olson discusses segment analysis and feature depth, pointing out that users typically utilize only 10 to 15 percent of a product's features. Host scores are 0 during presentation.8:56–11:18 · Matt as informed peer 0/10 Metric 4 - Efficiency of User Actions Olson details efficiency as a metric, using tax software to show that minimizing time-to-value matters more than usage frequency. Host scores are 0 for monologue.11:18–14:24 · Matt as informed peer 0/10 Metric 5 - Satisfaction and Qualitative Feedback Olson concludes his talk by exploring qualitative metrics like Net Promoter Score and emphasizing goal-driven measurement. Host scores are 0 for monologue.14:24–20:42 · Matt as informed peer 3/10 Q&A and Discussion on Product Leadership The host moderates a Q&A session asking about CPO organizational roles, company history, and instrumentation effort. Olson offers clear expert answers in an entirely non-adversarial dialogue.0:27–3:32 · Guest teaching 4/10 Defining What Makes a Great Product Todd Olson presents on product quality, explaining that traditional metrics like Google Analytics miss the mark when measuring product experience. Host scores are 0 as this is a solo presentation.3:32–5:57 · Guest teaching 5/10 The 5 Key Measurements Framework Olson outlines the first two metrics, breadth and frequency, explaining why accounts with under two active users pose severe churn risks in B2B SaaS. Host scores remain 0 for monologue.5:57–8:56 · Guest teaching 5/10 Importance of Segment Analysis Olson discusses segment analysis and feature depth, pointing out that users typically utilize only 10 to 15 percent of a product's features. Host scores are 0 during presentation.8:56–11:18 · Guest teaching 5/10 Metric 4 - Efficiency of User Actions Olson details efficiency as a metric, using tax software to show that minimizing time-to-value matters more than usage frequency. Host scores are 0 for monologue.11:18–14:24 · Guest teaching 4/10 Metric 5 - Satisfaction and Qualitative Feedback Olson concludes his talk by exploring qualitative metrics like Net Promoter Score and emphasizing goal-driven measurement. Host scores are 0 for monologue.14:24–20:42 · Guest teaching 5/10 Q&A and Discussion on Product Leadership The host moderates a Q&A session asking about CPO organizational roles, company history, and instrumentation effort. Olson offers clear expert answers in an entirely non-adversarial dialogue.0:27–3:32 · Guest disagreement 1/10 Defining What Makes a Great Product Todd Olson presents on product quality, explaining that traditional metrics like Google Analytics miss the mark when measuring product experience. Host scores are 0 as this is a solo presentation.3:32–5:57 · Guest disagreement 0/10 The 5 Key Measurements Framework Olson outlines the first two metrics, breadth and frequency, explaining why accounts with under two active users pose severe churn risks in B2B SaaS. Host scores remain 0 for monologue.5:57–8:56 · Guest disagreement 0/10 Importance of Segment Analysis Olson discusses segment analysis and feature depth, pointing out that users typically utilize only 10 to 15 percent of a product's features. Host scores are 0 during presentation.8:56–11:18 · Guest disagreement 0/10 Metric 4 - Efficiency of User Actions Olson details efficiency as a metric, using tax software to show that minimizing time-to-value matters more than usage frequency. Host scores are 0 for monologue.11:18–14:24 · Guest disagreement 0/10 Metric 5 - Satisfaction and Qualitative Feedback Olson concludes his talk by exploring qualitative metrics like Net Promoter Score and emphasizing goal-driven measurement. Host scores are 0 for monologue.14:24–20:42 · Guest disagreement 1/10 Q&A and Discussion on Product Leadership The host moderates a Q&A session asking about CPO organizational roles, company history, and instrumentation effort. Olson offers clear expert answers in an entirely non-adversarial dialogue.0:27–3:32 · Matt pushing back 0/10 Defining What Makes a Great Product Todd Olson presents on product quality, explaining that traditional metrics like Google Analytics miss the mark when measuring product experience. Host scores are 0 as this is a solo presentation.3:32–5:57 · Matt pushing back 0/10 The 5 Key Measurements Framework Olson outlines the first two metrics, breadth and frequency, explaining why accounts with under two active users pose severe churn risks in B2B SaaS. Host scores remain 0 for monologue.5:57–8:56 · Matt pushing back 0/10 Importance of Segment Analysis Olson discusses segment analysis and feature depth, pointing out that users typically utilize only 10 to 15 percent of a product's features. Host scores are 0 during presentation.8:56–11:18 · Matt pushing back 0/10 Metric 4 - Efficiency of User Actions Olson details efficiency as a metric, using tax software to show that minimizing time-to-value matters more than usage frequency. Host scores are 0 for monologue.11:18–14:24 · Matt pushing back 0/10 Metric 5 - Satisfaction and Qualitative Feedback Olson concludes his talk by exploring qualitative metrics like Net Promoter Score and emphasizing goal-driven measurement. Host scores are 0 for monologue.14:24–20:42 · Matt pushing back 1/10 Q&A and Discussion on Product Leadership The host moderates a Q&A session asking about CPO organizational roles, company history, and instrumentation effort. Olson offers clear expert answers in an entirely non-adversarial dialogue.

speaking balance: gold is Matt, purple is the guest (3 minute bins)

0:00 · Matt 100% · guest 0%0:00 · Matt 100% · guest 0%3:00 · Matt 100% · guest 0%3:00 · Matt 100% · guest 0%6:00 · Matt 100% · guest 0%6:00 · Matt 100% · guest 0%9:00 · Matt 100% · guest 0%9:00 · Matt 100% · guest 0%12:00 · Matt 84.2% · guest 15.8%12:00 · Matt 84.2% · guest 15.8%15:00 · Matt 73.1% · guest 26.9%15:00 · Matt 73.1% · guest 26.9%18:00 · Matt 92.6% · guest 7.4%18:00 · Matt 92.6% · guest 7.4%
Sharpest disagreement ▶ 1:45 Critiquing Google Analytics misusage

Olson mildly pushes back against standard industry practices, explaining how teams misconfigure Google Analytics and end up off by factors of 10.

 Hardest push from Matt ▶ 14:27 Questioning CPO adoption in enterprise

The host questions whether large Fortune 1000 enterprises actually embrace having dedicated product officers or still require evangelism.

 Biggest teaching moment ▶ 14:56 Explaining CPO vs CTO organizational dynamics

Olson educates the audience and host on why CPOs should report directly to the CEO to create necessary balance against CTOs who tend to say no.

 Matt holds his own ▶ 19:05 Reframing buyer value by leverage

When asked if Customer Success or CPOs buy Pendo, Olson clearly articulates why product leaders are a much higher leverage department to focus on.
the scores for every segment, with the reasoning behind each
ChapterTopicMatt as informed peerGuest teachingGuest disagreementMatt pushing backWhy
Defining What Makes a Great Product 0410 Todd Olson presents on product quality, explaining that traditional metrics like Google Analytics miss the mark when measuring product experience. Host scores are 0 as this is a solo presentation.
The 5 Key Measurements Framework 0500 Olson outlines the first two metrics, breadth and frequency, explaining why accounts with under two active users pose severe churn risks in B2B SaaS. Host scores remain 0 for monologue.
Importance of Segment Analysis 0500 Olson discusses segment analysis and feature depth, pointing out that users typically utilize only 10 to 15 percent of a product's features. Host scores are 0 during presentation.
Metric 4 - Efficiency of User Actions 0500 Olson details efficiency as a metric, using tax software to show that minimizing time-to-value matters more than usage frequency. Host scores are 0 for monologue.
Metric 5 - Satisfaction and Qualitative Feedback 0400 Olson concludes his talk by exploring qualitative metrics like Net Promoter Score and emphasizing goal-driven measurement. Host scores are 0 for monologue.
Q&A and Discussion on Product Leadership 3511 The host moderates a Q&A session asking about CPO organizational roles, company history, and instrumentation effort. Olson offers clear expert answers in an entirely non-adversarial dialogue.

Statements from this episode (4)

Insight
Olson: Revenue is a lagging indicator and insufficient for immediate product action
“The challenge with revenue is that it's a lagging indicator. So by the time you lose it is probably too late. Sorry if that's what your measure is, it's not enough to actually take action.”
Todd Olson Sep 28, 2017 ▶ 3:21
Insight
Olson: Software products typically rely on just five to ten value drivers
“If we have a hundred features, there's probably five to 10 value drivers that actually matter, that people really care about.”
Todd Olson Sep 28, 2017 ▶ 8:02
Disclosure
Olson: Pendo flags accounts as at-risk if they miss five key features
“For us, we have identified five right now, and those five are not just product key features, but our customer success team is also armed with them. Like, we know on our dashboards, if someone's not doing one of these five things, let's get flagged, we consider…”
Todd Olson Sep 28, 2017 ▶ 8:21
Insight
Olson: Splitting CPO and CTO roles creates beneficial organizational tension
“When you have a CTO separate from a CPO, it creates this natural tension where you have one person whose job is to kind of be driving the business, driving the strategy, and then the CTO is more focused on how it's done and also making sure that You know, I ca…”
Todd Olson Sep 28, 2017 ▶ 15:13
Made with StarZero

Turn any episode into a week of clips.

This entire site, over 400 conversations transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Try the podcast clipper Learn more