Experimentation
topic on 5 shows · 10 statements across 10 episodes
In Depth 
the Knowledge Project 
Lenny's Podcast 
the Neon Show 
the a16z Podcast
10 statements about Experimentation, every show
LaPointe: Teams stuck translating ideas instead of experimenting face market disadvantages
“And that team, because they got stuck in the translation phase instead of the experimentation phase, has a huge disadvantage in the market.”
Johari: Experimentation cultures foster risk aversion and overly incremental testing
“What I generally believe is that we're risk averse on both these two dimensions. That what people decide to test in a world that has promoted experimentation for everything tends to be more incremental by design. Okay. Because, and we'll come back to by actual…”
Ries: Respect for craft increases the necessity of testing everything else
“Having a respect for craft, for design, for perfection, that increases the need for experimentation and testing for all the other stuff.”
Weiss: Consistently neutral A/B test results signal a local optimum
“And this is, I think, often a good sign that you need to stop relying on experimentation and step back, which is that every experiment wasn't really positive or wasn't really negative. It was really just neutral. And it became increasingly hard to figure out W…”
Isford: Growth teams should use experimentation for risk mitigation, not optimization
“Generally my advice is to experiment when you need to, and to primarily see it as a risk mitigation tactic when you're making dramatic changes and to let the product development process do more work.”
Growth experimentation generates organizational learnings rather than delivering immediate outcomes
“And if you think about experimentation, It's not about delivering outcomes. It's about generating learnings that the organization can leverage effectively to deliver outcomes.”
Goel: Experimental corporate innovation programs must have high failure rates
“Everything we do in those programs they're supposed to have a high death rate. Right. If they don't, then we're just not doing enough. Like we're not doing too many, we're taking, we're not taking enough risk in experimenting.”
Frank Chen: Running small market experiments beats relying on predictive simulation
“It's hard to imagine a better system than the one we have, which is let's spend a little money and run a ton of experiments. Exactly. On businesses to figure out what people want. Because until you have it in the world, you're not sure what people will want.”
Robinson: Doubling down on failing methods fails; progress requires detached experimentation
“We think that what we're doing, we're not trying, as I'm using air quotes now, trying hard enough, or we'll invest some more money, and that's not changing what you're doing, that's just doing more of the same, and the goal is to be detached and experimental, …”
Mauboussin: Reversible Experiments Help Overcome Managerial Resistance to Change
“Another card to play in that is experimentation. Say, let's try it this way for, you know, this subset or for this period of time. And if it works out great, and if it doesn't work out, we'll revert to what we were doing before. So just, if that's not too cost…”