A B Test
topic on 7 shows · 15 statements across 15 episodes
the Y Combinator Startup Podcast 
the Knowledge Project 
Masters of Scale 
Latent Space 
Lenny's Podcast 
the Neon Show 
20VC
14 statements about A B Test, every show
Multi-year roadmaps waste tech's core advantage of iterative learning
“I think there's a second failure mode, which is people sit down and write up these long roadmaps and strategic vision docs of what we'll do over the next two or three years that kind of loses the comparative advantage we have over every other industry, which i…”
Nejatian: A/B testing primarily serves as a safety blanket for tech management
“What, for most tech companies, AB tests have done have been the safety blanket for management.”
Weitzman: A/B Testing Is Abdicating Decision-Making to Users
“A-B tests are abdicating decision making to the user.”
Goyal: Offline Evals Are Hardest to Build but Most Efficient Feedback Loop
“Offline evals, which I think is what a lot of the debate was about are both the most challenging to build feedback loop and also the most efficient once built. And then I think AB tests are a little bit less challenging to build and a little bit less efficient…”
Zhuo: You cannot A/B test your way into a great product
“You just actually can't make a really great product by thinking you can AB test your way into it.”
Ng: Use A/B testing to hone gut instincts, not just pick variants
“I don't just use the result of A-B test to pick product A or product B. My team will often sit down and look carefully at the data to hone our instincts, to speed up, to improve the rate at which we're able to use the first tactic to make high quality decision…”
Chopra: Wingify's revenue growth strongly correlated with platform A/B test volume
“For example, in our case, we built an A-B testing tool and we knew very clearly that the number of A-B tests that are on our platform are highly correlated with our revenue growth.”
Baxter: Notes drop likes and reposts by 30% to 40% in internal A/B tests
“We saw more like 30 to 40% engagement rate drops for likes and reposts and an AB test we ran. When comparing showing a post with or without a note”
Le Nel: Good product managers don't need A/B testing
“If you're a good product manager, you don't need an A-B test. You should be able to know what is the right product.”
Carter: The biggest growth wins often follow failed A/B tests
“The biggest wins come right on the heels of a failed A-B test where, like, that's not what I was expecting to see, but based on what I am seeing now, I've learned something new about the customer, so now let's go run a second A-B test on the basis of that lear…”
Lemkin: Startups must always hire their first two sales reps together
“So you need to hire One rep and you've got to hire two because otherwise there's no AB test. You have to AB test humans. You have to AB test humans. You've got to hire two as hard as it is.”
Johari: Rewarding only winning experiments treats failed tests as wasted time
“If I reward you for shipping winners, then what I'm really telling you is all the time that you spent testing out failures was wasted time.”
Hockenmaier: Monetization experiments require long-term five percent holdout groups
“Price and anything related to monetization is really tough, because it's basically a system of incentives, and it will change people's behavior. And so you have to observe these things over very long periods of time to make sure you're getting the right answer…”
Doshi: B2B startups struggle to run A/B tests compared to consumer apps
“If you're in consumer the benefit of being in consumer is that you generally have you usually will have like hundreds or thousands of users such that you could actually go do an A-B test, whereas in B to B it's a lot harder to achieve that.”