Aug 29, 2025 · 1h 18m · latent-space
Better Data is All You Need — Ari Morcos, Datology
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space podcast, Datology CEO Ari Morcos discusses why automated data curation is AI's most critical frontier, demonstrating how algorithmic filtering, synthetic rephrasing, and curriculum sequencing enable faster training, superior accuracy, and significantly smaller enterprise models.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 14% of the talking time here. How this is scored →
The hosts as informed peer 4.9 Guest teaching 5.4 Guest disagreement 2.6 The hosts pushing back 2.2
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 7:55 Calling Out Flawed Scaling Law Assumptions
Ari rejects standard scaling law literature from Kaplan and Chinchilla, calling the underlying assumption of IID data insane.
Hardest push from the hosts ▶ 52:56 Swyx Challenges Pre-Training Feedback LoopSwyx directly presses Ari on how pre-training data can be optimized for downstream tuning if post-training is merely capability elicitation.
Biggest teaching moment ▶ 17:36 Human Intuition Failure in DCLM StudyAri educates the hosts on the DCLM study showing expert NLP researchers performed no better than chance at guessing data filtering decisions.
The host holds their own ▶ 1:00:45 Swyx on Lottery Ticket Pruning and Hardware BottlenecksSwyx brings up specific historical research on parameter pruning and Frankle's lottery ticket hypothesis, demonstrating detailed technical context.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Ari Morcos's Background: Neuroscience, Inductive Biases, and the Bitter Lesson | 4 | 6 | 2 | 1 | Swyx demonstrates technical familiarity with vision transformers and weight mapping between CNNs and ViTs. Ari walks through his transition from neuroscience to inductive biases and his realization that architecture matters far less than data scale. | |
| The Strategic Shift: Why Data Is AI's Most Under-Invested Frontier | 3 | 6 | 4 | 1 | Ari strongly criticizes foundational scaling papers by Kaplan and Chinchilla for assuming IID data distributions, calling the premise insane given standard computer science adages. The hosts listen attentively with minimal interjections. | |
| Historical Prejudices and Misaligned Incentives in Data Research | 5 | 5 | 3 | 2 | Alessio asks why data research incentives are misaligned given high-profile data companies like Scale AI. Ari differentiates academic research culture, which treated data as fixed grunt work, from industrial priorities and supervised dataset limitations. | |
| The Self-Supervised Paradigm Shift: Moving to the Underfitting Regime | 4 | 6 | 3 | 1 | Ari presents a contrarian view on transformer architectures, arguing they are just one of many viable architectures while self-supervised learning on unlabeled data was the true revolutionary breakthrough. | |
| Data Engineering at Frontier Labs and the Failure of Human Intuition | 5 | 7 | 4 | 2 | Swyx notes his belief in automated end-to-end learning. Ari backs this up with findings from DCLM showing domain-expert grad students could not predict data filtering classifiers better than chance, proving human intuition fails at scale. | |
| Concept Complexity, Data Redundancy, and Zero-Shot Generalization | 4 | 6 | 2 | 3 | Alessio presses Ari on how to empirically define boundaries for concepts. Ari uses an analogy comparing elephants and dog breeds to explain why redundancy requirements vary with semantic complexity. | |
| The Evolution of Public Datasets, Code Predictors, and Copyright Litigation | 6 | 5 | 2 | 2 | Swyx rapid-fires standard pre-training datasets including GitHub, Arxiv, and Books3, citing litigation against Anthropic and Meta. Ari notes counterintuitive findings from StarCoder showing GitHub stars do not predict code quality. | |
| Overcoming the Limits of Power-Law Scaling via Data Efficiency | 4 | 6 | 3 | 2 | Ari explains the mathematical connection between decaying marginal information gain and power-law scaling, citing his NeurIPS paper on bending scaling curves through active data pruning. | |
| Quantifying Data Curation Gains Across Speed, Quality, and Size | 6 | 6 | 3 | 5 | Swyx challenges whether Datology's performance claims are simply overfitting or training to the test on public benchmarks. Ari details their strict protocol using held-out evaluations and unreleased eval suites. | |
| Why Specialized Data Companies Outperform Internal AI Lab Teams | 5 | 4 | 2 | 3 | Alessio questions whether private curation layers create friction with open-source dataset creators. Ari details Datology's moat balance between open scientific intuition and proprietary engineering know-how. | |
| Curation Methodologies and the Power of Synthetic Data Rephrasing | 6 | 5 | 2 | 2 | Swyx explores synthetic data, drawing distinctions between distillation and steganography. Ari outlines Datology's core approach of rephrasing existing data to bypass mode collapse and teacher model limitations. | |
| The Supremacy of Data Diversity and the Reality of Epoching | 5 | 5 | 3 | 2 | Swyx brings up the 'Textbooks Are All You Need' hypothesis. Ari refutes narrow distribution claims, asserting data diversity is paramount and epoching high-quality tokens consistently outperforms low-grade new data. | |
| The Resurgence of Curriculum Learning in Modern LLM Training | 4 | 6 | 2 | 2 | Ari explains the conceptual graph theory behind curriculum learning, clarifying that discrete curricula now succeed in underfitted LLM regimes where they previously showed minimal value in saturated supervised settings. | |
| Optimizing Pre-Training Data for Downstream Post-Training and Alignment | 6 | 6 | 4 | 5 | Swyx challenges the idea of pre-training dependencies given the consensus that post-training is merely capability elicitation. Ari counters that pre-training determines the slope of test-time compute scaling and alignment robustness. | |
| Model Pruning Bottlenecks and Complementary Optimization Strategies | 7 | 4 | 2 | 2 | Swyx brings deep context regarding Jonathan Frankle's lottery ticket hypothesis and parameter pruning. Ari explains how unstructured pruning fell out of favor due to GPU sparse matrix multiply inefficiencies. | |
| Future Architecture: Small Models, Test-Time Compute, and Cognitive Cores | 6 | 5 | 2 | 2 | Swyx references Andre Karpathy's 'cognitive core' concept. Ari agrees that storing raw knowledge inside network parameters is inefficient and smaller models will dominate test-time compute workloads. | |
| Case Study: The Arcee Foundation Model and Compounding Curation Gains | 5 | 5 | 1 | 1 | Alessio asks for concrete metrics from the Arcee training run. Ari details how condensing 25T tokens into 7T allowed a 4.5B model to beat larger baselines before reaching one trillion tokens. | |
| Data Valuation: AI's NP-Complete Problem and the Datology Profile | 4 | 5 | 2 | 1 | Swyx asks recruiting and technical frontier questions. Ari defines target-specific data valuation as AI's central NP-complete problem and outlines Datology's mission to automate zero-shot domain adaptation. |
Statements from this episode (50)
Morcos: Data curation choices fundamentally determine machine learning model performance
“There are a ton of choices you would make in that process, ranging from how you're going to filter the data, how you're going to sequence the data, what synthetic data you're going to generate, if any how you're going to batch the data, all of those things. An…”
Morcos: Proper data curation enables smaller models with equal or better performance
“Help the folks we work with to train models much faster to much better performance and to also help them train much smaller models to the same or better performance, which I actually think is some of the most exciting stuff going forward. But fundamentally, th…”
Morcos: Soft inductive biases become harmful past 1M data points in vision
“Turns out in the small data regime, and when I say small data here, I mean, say less than 500,000 data points. And this was in the context of image self-supervised learning. So in that small data regime, this is super helpful. And where this paper's actually b…”
Morcos: Inductive biases matter not at all at scale compared to data
“Basically, you know, when you get to enough scale, inductive biases matter not at all. All that really matters is the learned posterior from the data distribution, and that's really what defines everything.”
Morcos: Data is AI's most under-invested research area relative to impact
“Something I've said before and I'll say again is, is that data is the most under-invested in area of research relative to its impact, and I don't think it's even close.”
Morcos: Kaplan and Chinchilla scaling laws incorrectly assume all data is equal
“And even if you go and you look at the scaling laws work from Kaplan and Chinchilla and all these other things, they all assume IID data which is insane. We know that all data are not created equal, that garbage in garbage out is like the oldest adage in compu…”
Morcos: Industry values data work far more than academic AI research
“In general data work has been far more valued in industry consistently than it had been in the research community.”
Morcos: Top AI researchers' secret to success is looking at the data
“If you talk to the most talented AI researchers and you ask them, what's the secret to your success, they'll largely tell you that they look at the data.”
Morcos: The Transformer is just one of many equivalently good architectures
“And one of my like more controversial viewpoints, I think, is that I think the transformer is a great advance to be sure, but I think it's one of a very large Set of equivalently good architectures that we could have found. And there are many, many ways we cou…”
Morcos: Modern AI capabilities depended entirely on self-supervised learning
“But I do not think there's any way we could get to where we are today without self-supervised learning and the ability to train on unlabeled data. That was the real advance to my mind that enabled us to get these incredible increases in capabilities.”
Morcos: Better data improves AI performance per dollar by orders of magnitude
“Making the data better can be a massive compute multiplier. It can change the performance per dollar by orders of magnitude.”
Morcos: Data curation requires compounding dozens of individually modest, conflicting techniques
“Data creation also is a hard problem to solve quote unquote, because it's not one where there's a single silver bullet. There's not just do this one trick and all of a sudden things work. It's rather here are these 50 different things that you can do, each of …”
Morcos: DCLM researchers could not predict their own classifiers' filtering decisions above chance
“These are nominally the best experts you could ever hire to do this. These are students who have just spent all of their time looking at NLP data for two years. They could not predict what the DCLM classifiers would say above chance.”
Morcos: A data point's value depends on its relationship to the full dataset
“The easiest way to think about this is that the value of a data point is not just a function of that data point itself. It's rather a function of how that data point relates to every other data point in the training set, right?”
Morcos: Complex, high-variance concepts require much more data redundancy than simple ones
“The amount of data that I need in order to properly understand dogs is going to be a lot higher than the amount of data I need to understand elephants.”
Morcos: GitHub stars do not predict code quality for model training
“Stars are not a good predictor of whether data is useful for models or not. Like, I think that's, like, the most popular repos are not necessarily higher quality, at least with respect to do they improve a model's coding capabilities.”
Morcos: Zuckerberg personally approved high-risk AI training datasets at Meta
“When I was at Meta, certainly legal stuff around data sets was very challenging and becoming increasingly challenging, and there are a number of situations where, you know, the only person that could approve things was Zuck because of the scale of the risk, I …”
Morcos: Power-law scaling yields diminishing returns for every 10x data increase
“Power law scaling is terrible. It means that every time you 10 X your data, you get a diminishing marginal return on performance.”
Morcos: GPT-4.5 and Llama 4 show limits of naive mega-model scaling
“And I think that's what we've seen to some extent with the failure of the mega models, right? With 4.5 and Lama four and others. I think that there is a challenge of just continuing to do that naively and you have to figure out how to break it.”
Morcos: Proper data curation can bend neural scaling laws
“And what that paper showed was that if you use your data correctly, you can actually bend the scaling laws themselves.”
Morcos: Nemotron dataset quality is similar to DCLM despite token gains
“Nematron is actually pretty similar in quality to DCLM. It's, it came out about six months later. It has more unique tokens. They made a really big deal about it having more unique tokens, but on average, the quality is, is pretty straightforward.”
Morcos: Datology matches DCLM performance 12x faster with under 10% tokens
“We're able to now get to the same performance as DCLM about 12 x faster. So, you know, in fewer than 10% of the tokens we can match What you get from training to convergence.”
Morcos: Data curation still has at least 100x in performance gains ahead
“You know, we've already been able to get 10 X gains. I think there's at least another hundred X behind this that are still to be done.”
Morcos: Frontier AI lab data teams are systematically under-resourced
“I think you, what you see in all the frontier labs is that they have data teams. And if you talk to the folks that work on those data teams, what you'll kind of systematically hear is that typically they're under resourced relative to the gains that they're de…”
Morcos: Datology publishes intuition in blogs without enabling reproducibility
“What we've tried to do, and I think we've done a good job of, and I'm generally happy with the balance we've struck is try to, in the blog posts that we put out, give a lot of intuition as to kind of what we're doing and how it works without necessarily gettin…”
Morcos: Gemini tech report names data quality as single most important factor
“If you look at like the data section of like the Gemini tech report, it basically says like data quality was the single most important thing for making great model.”
Morcos: Post-training techniques are better applied in pre- and mid-training
“Most of what we do in post-training is better
were done in pre and mid training and earlier on in training in general.”
Morcos: Training a specialized frontier model will cost under $1M very soon
“I believe that getting to a frontier model should cost a million dollars or less for most organizations, at least in a specialized domain, right?
And when you think about what enterprises need, that's generally what they need.
They don't need a model that can …”
Morcos: Filtering synthetic data between generation cycles prevents model collapse
“If you filter the data after each point, that's now information injection, and that can break all of this and I think can prevent model collapse.”
Morcos: Weak models can rephrase data to train superior models
“Because the model that's doing the rephrasing just needs to know how to rephrase.
It doesn't need to know anything about the content itself.
It doesn't need to understand it.
It means you can use a pretty weak model.
To do the rephrasing and have it generalize…”
Morcos: Data diversity is the single most important factor in AI data quality
“If there's only one thing that you should take away from this entire interview about what is good for data quality, it's diversity.”
Morcos: Epoching high-quality data beats training on new average data
“Repeating higher quality tokens is almost always better than seeing net new lower quality tokens. So, like, epoching over higher quality data, almost always better than getting the same amount of new data of an unknown quality, or of average quality, average i…”
Morcos: Proper training curricula could reduce model training costs by 10x
“And getting a curriculum right could literally make the difference between, you know, spending 10 times as much on a model training, you know, hundreds of millions of dollars potentially.”
Morcos: Major AI labs fail to holistically integrate pre-, mid-, and post-training
“Something that you don't see happen even at the big labs because they have entirely separate teams, right? There's a free training team. There's a mid training team. There's a post training team. And like the mid training team is a customer of the free trainin…”
Morcos: Post-training alignment is ineffective long-term compared to pre-training alignment
“Like fundamentally, I think alignment and post training doesn't really make sense as a long-term solution. If you can easily align a model through post training, you can easily misalign a model through post training. If it's easy to put it in, it's easy to tak…”
Morcos: Qwen is much easier to align than Llama due to pre-training
“It's much easier to RL Quen than it is to do Lama. Likely that has to do with the fact that Quen put a lot of synthetic reasoning traces into their training data.”
Morcos: AI total cost of ownership is dominated by inference
“When you think about the total cost of ownership of these models, it's gonna be very, very heavily weighted towards inference. It's all inference.”
Morcos: AI inference costs will skyrocket, penalizing oversized models
“The inference costs are going to skyrocket with these models. And if you use a general purpose model, then you constrain to say, hey, this model knows about everything, but now only do this one thing. That model is going to have a ton of parameters that do not…”
Morcos: AI training is commoditized while data curation remains hard
“Mosaic was the first one to really recognize that there was a huge opportunity in making this easy. And now this has largely been commoditized by things like SageMaker and Together and lots of different folks that help you on the training side. But on the data…”
Morcos: Lottery ticket initializations fail because they are data dependent
“We actually found out that the problem was that the lottery ticket was actually data dependent. And that was where the fundamental problem came. That as soon as you change the data distribution a little bit, like the winning tickets changed in a really big way…”
Morcos: Most AI models used in three years will be under 10B parameters
“Most of the models that the vast majority of people will be using in say three years will be single digit B or smaller.”
Morcos: Test-time compute fundamentally favors smaller models to cut multi-step inference costs
“Test time compute as a paradigm really pushes you towards smaller models, right? Because if your cost of solving a problem is cost of inference times number of thinking steps, and you have to do a lot of thinking steps. Well, now this is like a really like min…”
Morcos: Current AI models waste massive capacity memorizing unnecessary knowledge
“We're wasting a ton of capacity in these models on knowledge that is just totally unnecessary for them to have.”
Morcos: Arcee 4.5B beat Gemma before reaching one trillion tokens
“It was beating Gemma pretty consistently before the one trillion mark, which was pretty cool to see.”
Morcos: Combining disparate data curation techniques generally fails without difficult tuning
“When you take these different techniques and you try to make them work together, they don't, generally. You can make them work together, but it's quite hard to do so.”
Morcos: Curation gains stack multiplicatively and preserve relative dataset advantages
“If we apply our curation on top of say DCLM, and then we apply it on top of FineWeb, the gap between FineWeb and DCLM is maintained in the gap between kind of Datology curated DCLM and Datology curated FineWeb. They both get a lot better, but Datology DCLM is …”
Morcos: No universal 'golden' curation exists for AI training data
“There's no golden curation. A curation is only optimal with respect to a given set of downstream use cases or tasks, right?”
Morcos: Valuing data for downstream use cases is AI's NP-complete problem
“In many ways, I think that's kind of the NP-complete problem of AI. If you can do that, you can kind of do anything”
Morcos: Yann LeCun was never defining Meta's AI strategy
“I don't think he was ever you know, or at least not since the beginning in a role where he was defining AI strategy for Meta. I don't think that's the role he wanted at any point. You know, I think he really wanted to be doing that research, and I think, so I …”
Morcos: Meta's metaverse bet will pay off in the long run
“I think the one that's still really up in the air is a metaverse, but I would actually argue that I think that's going to end up paying off in the long run. I think the Ray-Ban glasses pretty darn cool. And a lot of the foundations of what was in reality labs …”