Kevin Weil: A/B testing is orthogonal to product boldness
Kevin Weil · Kevin Weil: Lessons from Leading Product at Instagram & Twitter | 20VC #934 · 20VC with Harry Stebbings · Oct 7, 2022 · at 10:58
Kevin Weil, former Head of Product at Instagram and Twitter, discusses how product teams can run ambitious feature rollouts while remaining data-driven through testing.
0:00 / 0:43exact quote · 43.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Testing doesn't mean timidity. You can always use the fact that you're testing something and you can run an AB test to test something, 99% and have a one percent hold back. You don't have to test one percent in Australia, right? Testing gives you data. You can, if you believe in what you're doing, you can test by rolling it out to 99%. You can also test something aggressive, like putting stories at the front of the at the top of the user experience, and you can test that in Australia, but you don't have to test something on the third tab, you know, underneath the scroll. You can still test something really ambitious. So testing is a way to understand what you're building and how it's working for people, and that's orthogonal to whether what you're testing is bold or not.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Kevin Weil
Weil: Instagram Stories would have failed with a timid, experimental rollout
“If instead we'd done the timid thing and put it like in the back of a third tab and like, well, if people find it, we'll do a one percent experiment and maybe it'll work. Like it never would have taken off.”
Weil: Directly plopping Snapchat Stories into Instagram would have failed
“But I think if you had just taken stories and like plopped it in the middle of Instagram, it wouldn't have worked.”
US State Department asked Twitter to reschedule maintenance during Arab Spring
“Someone from the State Department reached out, and this is a public story, so I'm not saying anything that can't be said. Someone from the State Department reached out and said, wait, wait, you can't go down at, you know, two a.m. Pacific. That's the middle of…”
Weil: Effective organizations alternate leadership tiers between instigators and harmonizers
“You want alternating skills, just like you were saying, you want alternating skills, and he kind of classified people into, like, you know, are they kind of I'm trying to remember the word he used I don't remember, he had a provocative word for it, but basical…”
Weil passed on investing in former colleagues who built successful startups
“There are a couple of companies that I chose not to invest in because they were founded by people whose work that I just, that, that I worked with and for whatever reason in that environment weren't amazing. And so I made the assumption that that was them, not…”
Weil: Venture capital industry is fifty-fifty on hard work versus hype
“Less hype, more hard work for founders. I think the industry's kind of fifty-fifty there, you know. Some people are in, like, hustling every single day for the right things and helping their portfolio companies, and others are not.”