Weiss: Consistently neutral A/B test results signal a local optimum
Noah Desai Weiss · Lessons from Slack on decision making, product-led growth, and taking big swings | Noah Desai Weiss · Jul 13, 2023 · at 8:02
Noah Desai Weiss, Chief Product Officer at Slack, describes how Slack recognized that its onboarding funnel experiments had reached a plateau in mid-2020.
0:00 / 0:21exact quote · 21.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And this is, I think, often a good sign that you need to stop relying on experimentation and step back, which is that every experiment wasn't really positive or wasn't really negative. It was really just neutral. And it became increasingly hard to figure out What's going on where we can't make any positive or negative effect, which kind of seems to me that we had reached a, at least a local optima with the overall kind of end-to-end experience.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Noah Desai Weiss
Weiss: Data solves easy product problems, but big swings require intuition
“What we used to actually say early on was, you know, data can help solve easy problems, but doesn't actually solve the hard problems. And hard problems are ones where you can't actually just experiment your way, you know, down incremental paths. You actually h…”
Weiss: Product taste is not innate and can be 100% learned
“My honest kind of feeling is that almost everything can be taught. That teaching may be more informal. It may be more observational. You know, product design, product intuition, I think is really about seeing what great looks like and being part of the process…”
Weiss: Next-gen product-led SaaS companies will reach $10B within a decade
“And then I think kind of the next generation of companies like notion air table and so on, you know, these companies are all early on in their life cycle. I think give it another decade. And I think, you know, the companies that are a billion or two billion mi…”
Weiss: Product managers acting like "mini-CEOs" alienate their teams
“I will tell you that people who take the approach of being a mini CEO for a team are the ones who their teams wind up saying, why do we have to have a product manager on our team? Can't we do it without them? Because, you know, you can wind up accidentally bei…”
Weiss: PLG companies expanding to enterprise shouldn't start with outbound BDRs
“Starting off with building out a huge BDR team to do inside sales for cold leads feels like If you already had a product-led growth motion, not the right place to begin. Because I think, without a doubt, the best leads aren't ones that you cold emailed or that…”
Weiss: Product founder CEOs should follow a U-shaped involvement model
“If you think about like a major product initiative, not an experiment, but something more significant, that's like this big kind of bet. And you think about the X axis is kind of the time that goes on from the project starting to maybe launch. And the Y axis i…”