Hockenmaier: High-effect experimentation yields faster learnings than pure testing velocity
Dan Hockenmaier · Advice on Organizing and Running Growth Teams from Dan Hockenmaier and Gustaf Alströmer · Y Combinator · Jun 26, 2019 · at 11:53
Dan Hockenmaier explains that growth teams should prioritize testing fundamentally different hypotheses to generate large effect sizes rather than rushing through minor tactical experiments.
0:00 / 1:05exact quote · 65.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I also think, like, one of the mistakes that companies make a lot is this idea that speed of testing or velocity of testing is the most important thing. It is important to move fast because it is quite hard to have good hypotheses about what's going to work. Like, I am constantly wrong about trying to predict the results of experiments. And so you do need to just test a lot of things. However, if you just get, if you optimize for speed at all costs you know, you should be instead thinking about things like how do we test really disparate things to make sure that we are really testing at the ends of the experience to understand, you know, what's going to drive. So instead of like moving minor things around the conversion flow, testing entirely different flow or a very different experience not only will you learn more you know, based on kind of where the, Result, and that's out, but you actually will get results back faster, because sample size is a product of your effect size, and so, like, driving a higher effect is more important, and I think the other thing is, if you, like, look at a backlog of 80 or a hundred experiments, often there's only four or six hypotheses underneath that, and you should really be, like, thinking about which hypotheses you're testing, rather than just trying to run through all 80 tests”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dan Hockenmaier
Hockenmaier: Early big tech growth playbooks mislead startups without product-market fit
“I actually think like the legacy of that growth team, and then like Twitter and LinkedIn and the others that followed is actually maybe problematic when we think about some of the applications of growth later on, because they were applied in this case of like …”
Hockenmaier: Startups should adopt third-party experimentation tools rather than building internally
“I think increasingly the third party tools to do this The full stack tools are pretty good, actually. And so even for quite talented teams, the answer is usually not to build it yourself right away. You know, I'd love to hear if you have a rule of thumb on thi…”
Hockenmaier: Startups should keep growth in-house to preserve rich customer feedback
“For almost all things related to growth, paid marketing not only do you want to build that muscle yourself, but it's like the front line. You're dealing with your customers and the feedback is so rich that if that person is not sitting next to somebody else on…”
Hockenmaier: 90% of growth advising is talking founders out of ideas
“Nine times out of 10, we are talking them out of doing things that they want to do.”
Hockenmaier: Startups should not scale acquisition before cohort retention curves flatten
“You don't have to have best in class retention numbers, but you need to be seeing some cohorts of people for which retention truly levels out and they use the product very longterm. And if you don't see that you're not ready.”