Production A/B testing at Yahoo and Facebook is a costly experiment
Raymie Stata · Becoming an Internet Company // Raymie Stata, Altiscale [FirstMark's Data Driven] · Dec 8, 2016 · at 9:55
Raymie Stata, founder and CEO of Altiscale and former Yahoo CTO, explains why companies should not treat production A/B testing as their primary or initial experimentation method.
0:00 / 0:25exact quote · 25.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“If you think about A-B testing, that lives much more at the top of the ladder than at the bottom. You know, to actually get to the point where you're, If you're doing A-B testing in production at a large company like Yahoo or Facebook, you have to implement the code. There's typically a lot of pre-screening that you have to do. You know, the buckets, so to speak, that people run these tests in are highly contended, so there's actually opportunity cost for running an A-B test. You know, it's a fairly expensive experiment to make.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Raymie Stata
Stata predicts IT infrastructure will move to the cloud by 2026
“My personal view is, you know, 10 years from now, where it's going is clear, and I think that, when I started the company, companies like Bloomberg, for example, would say, no way, no how, we'll never do anything like, you know, like, you know, big data in the…”
Stata predicts all companies will become internet companies
“I believe everybody, all companies are going to become internet companies.”
Stata: Perks like foosball and free food do not define internet companies
“What I do not mean is that, hey, install foosball tables, have free food, and growth will follow. That's not what defines an internet company, especially in this context.”
Stata: Internet company system architecture transfers directly to traditional enterprises
“Now when you, when the onstage kind of becomes buildings and printers and trucks, you know, my claim is that this architecture really doesn't change. And a lot of what we learned from an from an architecture and system design perspective will transfer fairly d…”
Stata: Sophisticated experimentation relies on ladders from cheap to definitive tests
“As you get more sophisticated at running experiments, what you do is you build these ladders of experiments, where at the bottom rungs of the ladder, the experiments are cheap you know, and, but the outcomes are a little bit more questionable. As you get to th…”
Stata: Enterprise transformation is social engineering, not technical engineering
“While that data infrastructure certainly is a bit of a blocker, right, to transforming your business, actually changing business processes is, is much harder, right? That's a social engineering exercise. It's not a technical engineering exercise, and those tak…”