Hockenmaier: Monetization experiments require long-term five percent holdout groups
Dan Hockenmaier · Advice on Organizing and Running Growth Teams from Dan Hockenmaier and Gustaf Alströmer · Y Combinator · Jun 26, 2019 · at 34:08
Dan Hockenmaier (founder of Basis One) outlines best practices for measuring long-term user behavior changes during pricing experiments.
0:00 / 0:25exact quote · 25.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Price and anything related to monetization is really tough, because it's basically a system of incentives, and it will change people's behavior. And so you have to observe these things over very long periods of time to make sure you're getting the right answer. And so I'm like a big fan of running A-B test for a sufficient amount of time and then flipping it to, you know, maybe five percent remaining in, in the original test for a very long term to observe the thing that you thought happened continues to happen and validating that.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dan Hockenmaier
Hockenmaier: Early big tech growth playbooks mislead startups without product-market fit
“I actually think like the legacy of that growth team, and then like Twitter and LinkedIn and the others that followed is actually maybe problematic when we think about some of the applications of growth later on, because they were applied in this case of like …”
Hockenmaier: High-effect experimentation yields faster learnings than pure testing velocity
“I also think, like, one of the mistakes that companies make a lot is this idea that speed of testing or velocity of testing is the most important thing. It is important to move fast because it is quite hard to have good hypotheses about what's going to work. L…”
Hockenmaier: Startups should adopt third-party experimentation tools rather than building internally
“I think increasingly the third party tools to do this The full stack tools are pretty good, actually. And so even for quite talented teams, the answer is usually not to build it yourself right away. You know, I'd love to hear if you have a rule of thumb on thi…”
Hockenmaier: Startups should keep growth in-house to preserve rich customer feedback
“For almost all things related to growth, paid marketing not only do you want to build that muscle yourself, but it's like the front line. You're dealing with your customers and the feedback is so rich that if that person is not sitting next to somebody else on…”
Hockenmaier: 90% of growth advising is talking founders out of ideas
“Nine times out of 10, we are talking them out of doing things that they want to do.”
Hockenmaier: Startups should not scale acquisition before cohort retention curves flatten
“You don't have to have best in class retention numbers, but you need to be seeing some cohorts of people for which retention truly levels out and they use the product very longterm. And if you don't see that you're not ready.”