Sample Ratio Mismatch

topic on 1 show · 2 statements across 1 episodes

Lenny's Podcast

2 statements about Sample Ratio Mismatch, every show

LENNY'S PODCAST Opinion

Bots are the most common cause of sample ratio mismatches in experiments

“So when you say most common, I think the most common is bots.”

Ronny Kohavi Jul 27, 2023 ▶ 58:40 The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

LENNY'S PODCAST Assertion Partly supported

Roughly 8% of Microsoft's online experiments had sample ratio mismatches

“And where we share that at Microsoft, even though we'd be running experiments for a while is around eight percent of experiments that suffered from the sample ratio of dispatch.”

Ronny Kohavi Jul 27, 2023 ▶ 57:19 The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

← every entity, every show

Turn any episode into a week of clips.

This entire site, thousands of episodes across every show transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Sample Ratio Mismatch

2 statements about Sample Ratio Mismatch, every show

Turn any episode into a week of clips.