Morcos: Soft inductive biases become harmful past 1M data points in vision
Ari Morcos · Better Data is All You Need — Ari Morcos, Datology · Aug 29, 2025 · at 6:04
Ari Morcos, CEO of Datology and former AI researcher at Meta and Google DeepMind, discusses empirical findings from his ConViT research on inductive biases versus dataset scaling.
0:00 / 0:46exact quote · 46.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Turns out in the small data regime, and when I say small data here, I mean, say less than 500,000 data points. And this was in the context of image self-supervised learning. So in that small data regime, this is super helpful. And where this paper's actually been cited is a whole bunch of kind of niche scientific problems where there's very little data, for example, volcano prediction where you have like 1500 data points or things like that. But the advantage of using this soft inductive bias decays as the data size increases and eventually actually becomes harmful. So if you see enough data and the threshold at which this changes is around like a million data points. So it's not massive by any stretch by our current model. So basically once you get past a million data points, that soft inductive bias no longer helps you. And it actually now is mildly harmful.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ari Morcos
Morcos: The Transformer is just one of many equivalently good architectures
“And one of my like more controversial viewpoints, I think, is that I think the transformer is a great advance to be sure, but I think it's one of a very large Set of equivalently good architectures that we could have found. And there are many, many ways we cou…”
Morcos: GPT-4.5 and Llama 4 show limits of naive mega-model scaling
“And I think that's what we've seen to some extent with the failure of the mega models, right? With 4.5 and Lama four and others. I think that there is a challenge of just continuing to do that naively and you have to figure out how to break it.”
Morcos: Post-training alignment is ineffective long-term compared to pre-training alignment
“Like fundamentally, I think alignment and post training doesn't really make sense as a long-term solution. If you can easily align a model through post training, you can easily misalign a model through post training. If it's easy to put it in, it's easy to tak…”
Morcos: Inductive biases matter not at all at scale compared to data
“Basically, you know, when you get to enough scale, inductive biases matter not at all. All that really matters is the learned posterior from the data distribution, and that's really what defines everything.”
Morcos: Data is AI's most under-invested research area relative to impact
“Something I've said before and I'll say again is, is that data is the most under-invested in area of research relative to its impact, and I don't think it's even close.”
Morcos: Kaplan and Chinchilla scaling laws incorrectly assume all data is equal
“And even if you go and you look at the scaling laws work from Kaplan and Chinchilla and all these other things, they all assume IID data which is insane. We know that all data are not created equal, that garbage in garbage out is like the oldest adage in compu…”