Ronny Kohavi
14 statements across 1 episodes · 4 bullish · 1 bearish · 1 people on the record · first statement Jul 27, 2023 by Ronny Kohavi · said 3 times in 3 episodes since 2023 · across every show →
On the record as a speaker too: Ronny Kohavi's record, appearances and statements → this page counts the times other people say the name.
Mentions by year
brought up most by Ramesh Johari (1), Lenny Rachitsky (1), Itamar Gilad (1)
tap a year for its mentions
2023 3 mentions in 3 episodes 1 per episode
Everything said about Ronny Kohavi, oldest first
Jul 27, 2023 neutral
Jul 27, 2023 positive
Airbnb's search team drove a 6% revenue gain across 250 experiments
“Another example that I am allowed to speak about from Airbnb Is the fact that we ran some 250 experiments in my tenure there in search relevance. And again, small improvements added up. So this became overall a six percent improvement to revenue.”
Jul 27, 2023 neutral
Jul 27, 2023 positive
Jul 27, 2023 neutral
Airbnb's search relevance team launched nothing without an A/B test
“One is in my team in search relevance, everything was A-B tested. So while Brian can focus on some of the design aspects, the people who are actually doing, you know, the neural networks and the search, Everything was AB tested to help. So nothing was launchin…”
Jul 27, 2023 neutral
Jul 27, 2023 neutral
Jul 27, 2023
Bundling multiple redesign changes is more likely to fail than incremental testing
“If you believe in that statistics that I published, then doing 17 changes together is more likely to be negative. Do them in smaller increments. Learn from, it's called OFAT, one factor at a time. Do one factor, learn from it, and adjust. Of the 17, maybe you …”
Jul 27, 2023 neutral
Jul 27, 2023 positive
Bing's search relevance team targets 2% annual compounding improvement
“One is at Bing, the relevance team. Hundreds of people all working to improve Bing relevance. They have a metric. We'll talk about, oh, we see the overall evaluation criterion, but they have a metric that their goal is to improve it by two percent every year. …”
Jul 27, 2023 positive
Automated scorecards are the primary way to accelerate experimentation speed
“One is if your platform is good, then when the experiment finishes, you should have a scorecard soon after. Maybe it takes a day, but it shouldn't be that you have to wait a week for the data scientists. To me, this is the number one way to speed up things.”
Jul 27, 2023 negative
Never ship flat experiment results due to the hidden maintenance overhead
“Flat to me, if something is not Statsig, that's a no ship because you've just introduced more code. There is a maintenance overhead. To shipping your stuff. I've heard people say, look, we already spent all this time. The team will be demotivated if we don't s…”
Jul 27, 2023
Airbnb required teams to replicate A/B tests with borderline p-values
“When I worked at Airbnb, one of the things we did is we said, okay, if you're less than .5, but above .1, rerun, replicate. When you replicate, you can combine the two experiments and get a combined p-value using something called Fisher's method or Stauffer's …”