Clowes: Atlassian kept 10% of users in a permanent global experiment holdout
Shaun Clowes · Why great AI products are all about the data | Shaun Clowes (CPO at Confluent) · Dec 29, 2024 · at 43:51
Shaun Clowes, former Head of Growth at Atlassian, explains how Atlassian measured long-term cohort baselines against experiments.
0:00 / 0:10exact quote · 10.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“At Adassian, we did something similar. We had a global holdout group, actually, that was held out of all experiments. There was an interview, experiment platform couldn't target that group at all. So, 10% of all people never saw anything ever.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Shaun Clowes
Clowes: AI will entrench dominant enterprise SaaS incumbents over startups
“The most likely outcome is that the currently dominant companies are going to get more dominant. But I don't think this idea that it would just cause a spring up of a whole bunch of new, new apps that will more easily challenge the incumbents makes any particu…”
Clowes: 10x product managers produce 100x return via leverage
“It must be true that if there's 10 times engineers, and I would argue there definitely are, there must be 10 times product managers too. But at the same time, those 10 times product managers, because product management is ultimately about leverage, so it's abo…”
Clowes: AI product quality depends on data context, not replaceable models
“What will make my AI experience really great? It's definitely not going to be the models because like these models are mostly going to be somewhat replaceable and you could say, okay, well, is it going to be the prompts? Maybe, but you know, so many good promp…”
Clowes: 90% of AI development effort goes into data management
“It's a data management problem. Like, it's getting access to good data, getting access to high quality data, getting access to timely data, and getting it to the LLM to get the LLM to make a smart decision. That's where 90% of the calories go.”
Clowes: Business rules, not UIs, drive enterprise SaaS lock-in
“The real thing, which is staring everybody in the face, is it's all about the business rules. Like, that is what drives the lock-in.”
Clowes: AI makes startup distribution harder by flooding channels with spam
“In the world of AI, it seems like distribution is more likely to get hard than easy. So, so like you know, if you think about, for example, diminishing returns on cold email, because cold email is getting easier and easier to send even worse spam. Like it soun…”