Johari: Standard frequentist A/B testing discards past experimental learning
Ramesh Johari · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor) · Nov 9, 2023 · at 55:44
Stanford Professor Ramesh Johari explains the statistical limitations of standard A/B testing frameworks in tech companies.
0:00 / 0:25exact quote · 25.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“You know, a funny thing about experiments is that we throw past learning away effectively. And this is just an artifact of how we analyze experiments that the methods used, the statistical methods used typically, you know, p-values, confidence intervals. These fall in a branch of statistics known as frequentist statistics. And the idea behind frequentist statistics, you know, without being overly technical, it's just, I let that data speak for itself. There's no beliefs brought to the table about where that data came from.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ramesh Johari
Johari: Marketplaces never start by solving two-sided matchmaking friction
“The moral is a marketplace business never starts as a marketplace business, because what we think of as a marketplace business is something which at scale is removing the friction of the two sides finding each other. But when you start, you don't have that sca…”
Johari: AI puts more pressure on human data scientists, not less
“What AI has done for us is it's massively expanded the frontier of things we could think about our problem, hypotheses we could have, maybe things we could test. It's just an astronomical explosion of explanations and ideas and principle. And I really think ac…”
Johari: OpenAI has evolved into a marketplace via plugins
“No one in their right mind would have thought of open AI as a marketplace. But OpenAI is a marketplace now. They may not want to call themselves a marketplace, but they have plugins. The plugins are flooding that, that platform.”
Johari: Virtually every modern tech business will have the option to become a marketplace
“Virtually every business is going to have that option at some point in, in, you know, the modern tech enabled economy anyway.”
Johari: Machine learning prediction is correlation, but business decisions require causation
“When we teach people to build machine learning models, we're asking them to make predictions. We're asking them to find correlations. Prediction is inherently about correlation. But when we ask people to make decisions, we're asking them to think about causati…”
Johari: Experimentation cultures foster risk aversion and overly incremental testing
“What I generally believe is that we're risk averse on both these two dimensions. That what people decide to test in a world that has promoted experimentation for everything tends to be more incremental by design. Okay. Because, and we'll come back to by actual…”