A/B testing is harmful unless statistical significance is reached within 36 hours
Rand Fishkin · Episode 537 | On Launching, Funding, and Growth with Serial SaaS Founder Rand Fishkin · Feb 23, 2021 · at 30:50
Rand Fishkin (co-founder of SparkToro, former CEO of Moz) discusses why early-stage startups should avoid split testing until they reach substantial traffic volume.
0:00 / 0:27exact quote · 27.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think A-B testing and conversion rate testing in general is great when you have between tens of thousands and hundreds of thousands or more visitors to a page and enough of them are converting that you can basically test in 12 to 36 hours. Anything under that and the testing itself will get in the way of you making real improvements.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Rand Fishkin
Venture capital only profits due to 1960s tax lobbying by the wealthy
“Venture capital is essentially an asset class that only makes money because rich people in the sixties and seventies lobbied the federal government to get capital gains tax rate applied to this certain type of investing, right?”
Investors avoiding dividend taxes are poor citizens and unwelcome at SparkToro
“If giving the government 12 to 20% of your money is unacceptable to you, first of all, I don't think you're a very good civic person. Right? Like, and second, I don't want you on my balance sheet anyway.”
Fishkin: Roughly 90% of venture-backed startup founders come from family wealth
“Like the whole idea behind venture capital is we are going to take a bunch of mostly already wealthy people because that's who starts venture backed startups, right? The overwhelming majority, I think something like 90% come from family wealth.”
TikTok organic reach will inevitably crash to near zero, mimicking Facebook
“And I think, unfortunately, a lot of these folks are going to find that the same thing that happened with Facebook seven years ago, six years ago, where that engagement rate falls from 11% to 0.09%, that will happen. That will happen to you.”
Moz's revenue growth sank significantly after Rand Fishkin stepped down
“After I stepped down as CEO, the growth rate sank significantly. It was trending a little bit down, like, the last six months I was there, so I wouldn't put the fault entirely on folks after I left, but, you know, the last, oh my gosh, the last seven years cer…”
Fishkin: About 20% of VC investments get stuck in growth purgatory
“Out of those, you know, average hundred investments that VCs make, the stuck in the middle is a really significant percentage. It's probably like 20 of those hundred, and a lot of those end up being some combination of, like, fire sales, or for more aggressive…”