Bing allowed experimenters to define up to 10,000 metrics per scorecard
Ronny Kohavi · The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon) · Jul 27, 2023 · at 1:10:55
Ronny Kohavi, former VP and Technical Fellow at Microsoft and Airbnb, discusses scaling experimentation platforms and managing computational load from large metric sets.
0:00 / 0:06exact quote · 6.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“At Bing, you could define 10,000 metrics that you wanted to be in your scorecard.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ronny Kohavi
Kohavi: Airbnb would be in a better state today with more experiments
“I believe that had Airbnb kept people like Greg really, which was pushing for a lot more data driven and had Airbnb run more experiments. It would have been in a better state than today, but it's the counterfactual. We don't know.”
Kohavi: Airbnb's revenue would have grown just as much without COVID pivots
“I think if Airbnb stayed the course, did nothing, the revenue would have gone up in the same way.”
Never ship flat experiment results due to the hidden maintenance overhead
“Flat to me, if something is not Statsig, that's a no ship because you've just introduced more code. There is a maintenance overhead. To shipping your stuff. I've heard people say, look, we already spent all this time. The team will be demotivated if we don't s…”
Airbnb's weak experimentation platform forced heavy hiring of data scientists
“I think, you know, one of the things that I will say at Airbnb is the analysis was relatively weak. And so lots of data scientists were hired to be able to compensate for the fact that the platform didn't do enough.”
Microsoft's overall experiment failure rate is 66%, reaching 85% at Bing
“So overall at Microsoft, About 66%, two-thirds of ideas fail, right? And don't think the 66 is accurate. Like, you know, it's about two-thirds. At Bing, which is a much more optimized domain after we've been optimizing it for a while, the failure rate was arou…”
Airbnb's experiment failure rate was 92%, while Google Ads hit 80-90%
“And then at Airbnb this 92% number is You know, the highest failure rate that I've observed. Now I've quoted other sources that, you know, it's not that I worked at groups that were particularly bad booking Google ads, other companies published numbers. There …”