A/B testing

“People often have this like, oh, you know, design and user experience is on the other side of the coin. You know, it's like, it's a totally different industry and they're at odds with each other, right? Being data informed and being quantitative versus like be…”

“There shouldn't be, especially in engineering within the world of growth, a culture around having to point to numbers to demonstrate your impact, because if there is, then the team will buy us to experiments all the time. And that's not necessarily the right t…”

“A-B testing is one of the most expensive kinds of ways to validate an experiment, right? It you know, often requires design and engineering and the PM or growth person or marketing person who's crafting it, right? All these things Or investments that take a lo…”

“The experiments are great, but they're also, they can be slow. You have to look at the results. You have to analyze how things went. You have to learn what's going on. You have to build the experiment. So there's a cost to an experiment and not everything need…”

“Unless you have at least tens of thousands of users, The math, the statistics just don't work out for most of the metrics that you're interested in.”

“So you ask for rule of thumb, 200,000 users, you're magical. Below that, start building the culture, start building the platform, start integrating, so that as you scale, you start to see the value.”

“Flat to me, if something is not Statsig, that's a no ship because you've just introduced more code. There is a maintenance overhead. To shipping your stuff. I've heard people say, look, we already spent all this time. The team will be demotivated if we don't s…”

“And I will say that nine out of 10, when we call out Twyman's Law, it is the case that we find some flaw in the experiment.”

“Cause to some degree, or maybe the way I think about experimentation, that's the highest bar that proves with near absolute certainty that there's a causal relationship between the change you made and the KPI that you want to move. But I think that that is, it…”

“We don't do AP testing or we don't do specific or follow like certain metrics or something. We might sometimes, we do have telemetry or like, we can look at like how people use certain things. And we sometimes Do that. But like, that's not usually the goal we …”

“You know, a funny thing about experiments is that we throw past learning away effectively. And this is just an artifact of how we analyze experiments that the methods used, the statistical methods used typically, you know, p-values, confidence intervals. These…”

“What I generally believe is that we're risk averse on both these two dimensions. That what people decide to test in a world that has promoted experimentation for everything tends to be more incremental by design. Okay. Because, and we'll come back to by actual…”

“AB tests are great, but one of my most painful things to do is to sit in a room full of PMs and data scientists who have just seen the results of an experiment that like flipped to Statsig. And then they're like, cool. I was significantly down over this course…”

“The best research to know a little bit of stats, like you can't interact in a world of A-B testing without knowing basic statistics.”

“If the intuition is right, that this probably is helping merchants, Why do we start with that? The original control is better if it's neutral. Let's start with like, what would we have shipped if we were a blank slate? And if it's neutral, actually neither is …”

“Engineers have a tendency to add these little incremental wins that actually ha add. You know, more of a long-term maintenance cost than is clear because you just run a little one month, AB test. You see this, you know, significant when they don't realize the …”

“People say the word eval is trying to kind of like carve out this new thing, and saying, you know, evals, and then A-B testing, but if you zoom out, it's the same data science as before, and I think that's what's causing the confusion is, hey, we need data sci…”

“Some places it used to take, you know, months to get something through production, to do A-B testing and get feedback. We can do this in a day or two, right? Definitely under a week.”

“You can't AB test the important things and the details are mostly false positives. So it's an enormous waste of time unless you're incredibly sophisticated.”