Dec 24, 2024 · 28m · latent-space
Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024]
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
At NeurIPS 2024, Loubna Ben Allal of Hugging Face presents a comprehensive overview of synthetic data generation pipelines and the rapid advancement of small, on-device language models. She demonstrates how synthetic pre-training, intelligent filtering, and compute-efficient over-training enable compact architectures like SmolLM2 to deliver high-performance, private, and cost-effective edge AI solutions.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 0.0 Guest teaching 0.0 Guest disagreement 0.5 The hosts pushing back 0.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 3:50 Refuting model collapse narratives
Loubna explicitly challenges prevalent media and Nature paper warnings about catastrophic model collapse, arguing empirical Common Crawl evidence shows synthetic-rich web data improves quality rather than degrades it.
Hardest push from the hosts ▶ 19:10 Challenging large model obsessionLoubna pushes back against the prevailing industry narrative that pursuing ever-larger models is optimal, emphasizing the steep recurring inference costs and showing that training smaller models on massive token counts yields superior trade-offs.
Biggest teaching moment ▶ 4:30 Explaining why small-scale collapse studies fail to generalizeLoubna breaks down why small-scale iterative self-training studies observe collapse whereas distilling knowledge from large, capable teacher models avoids quality degradation.
The host holds their own ▶ 27:00 Cyclical shift back to fine-tuningLoubna synthesizes the trajectory of modern NLP, articulating that after swinging from fine-tuning to massive model prompt engineering, cost pressures are driving engineering practice back toward domain fine-tuning of small models.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Opening Remarks and Talk Agenda | 0 | 0 | 0 | 0 | Solo presentation opening outlining the evolution of synthetic data from post-training to pre-training and evaluation. Host scores remain zero as this is a monologue presentation. | |
| Demystifying Model Collapse and Web Data Pollution | 0 | 0 | 2 | 0 | Loubna refutes conventional media and academic panic regarding model collapse and web pollution, presenting empirical data from FineWeb showing newer dumps perform better. | |
| Synthetic Pre-Training and the Cosmopedia Approach | 0 | 0 | 1 | 0 | Loubna discusses reproducing Microsoft's Phi methodology with Cosmopedia, explaining prompt seeding strategies to ensure synthetic data diversity. | |
| Web Rephrasing and Programmatic Data Refinement | 0 | 0 | 0 | 0 | Technical breakdown of web rephrasing techniques using small models and programmatic refinement from Nemotron CC and ProX. | |
| LLM-Based Quality Classifiers for Large-Scale Filtering | 0 | 0 | 0 | 0 | Detailed walkthrough of FineWeb-Edu and DCLM classifier-based quality filtering pipelines. | |
| Post-Training Advancements: AgentInstruct, TÜLU 3, and SmolTalk | 0 | 0 | 0 | 0 | Review of post-training datasets including AgentInstruct, TÜLU 3, SmolTalk, and multilingual data arbitrage. | |
| The Emergence of High-Performing Smol and Edge Models | 0 | 0 | 0 | 0 | Demonstration of competitive small models running on consumer hardware like iPhones via PocketPal. | |
| Rethinking Scaling Laws: Inference Economics and Long Training | 0 | 0 | 1 | 0 | Loubna challenges the 'bigger is always better' scaling paradigm, arguing for overtraining smaller architectures due to inference economics. | |
| On-Device Deployment, Tooling, and Structured Generation | 0 | 0 | 0 | 0 | Discussion of on-device advantages, privacy, and JSON-constrained structured text generation in browser. | |
| Future Trends, The Return of Fine-Tuning, and Q&A | 0 | 0 | 1 | 0 | Loubna concludes with a hot take predicting the industry will cycle back to fine-tuning specialized small models over prompt engineering. |
Statements from this episode (21)
Ben Allal: LLMs can be trained with entirely synthetic pipelines
“Today you can train an LLM with like an entirely synthetic pipeline. For example, you can use our Cosmopedia data sets and you can train a one B model on like a hundred and fifty billion tokens. Those are a hundred percent synthetic, and those are also of good…”
Ben Allal: Properly curated synthetic data prevents model collapse
“And I think there's a lot of concerns about model collapse, and I'm going to talk about that later, but we'll see that like, if we use synthetic data properly and we curate it carefully that shouldn't happen.”
Hugging Face finds LLM proxy words jumped in Common Crawl after ChatGPT
“For example, here we measured like these words ratio in different dumps of common crawl, and we can see that like the ratio really increased after chat GPT's release.”
Ben Allal: Recent web dumps improve model benchmarks despite synthetic data
“So what we did is we trained different models on these different dumps, and we then computed their performance on popular like NLP benchmarks, and then we computed the aggregated score. And surprisingly, you can see that the latest dumps are actually even bett…”
Ben Allal: Synthetic data may enrich the web rather than pollute it
“So personally, I wouldn't say the web is posted with synthetic data. Maybe it's even making it more rich.”
Ben Allal: Distilling from large models avoids small-model self-training collapse
“I think if you do that approach, it's normal to observe this kind of behavior because the quality is going to be worse because the model is already small, and then if you train it just on these generations, you shouldn't expect it to become better. But what we…”
Ben Allal: Prompt diversity is essential for scaling synthetic training data
“The key ingredient to getting a good data set that is synthetic is trying as much as possible to keep it diverse, because if you just throw the same prompts as your model, like generate, like, a textbook about linear algebra, and even if you change the tempera…”
Synthetic college text boosts MMLU; middle school text boosts OpenBookQA
“College textbooks are really good for MLU or middle school textbooks are good for benchmarks like open book, UA and Pico.”
Ben Allal: NVIDIA generated 1.9 trillion synthetic tokens for Nemotron-CC
“This is a recent paper from NVIDIA, Mnemotron CC. They took things a bit further and they generated not a few billion tokens, but 1.9 trillion tokens, which is huge.”
Ben Allal: Small models can generate synthetic data by rephrasing web pages
“The interesting thing in this approach is that you can use a model that is small Because it doesn't, rewriting doesn't require knowledge. It's just rewriting a page into a different style. So the model doesn't need to have like knowledge that is like extensive…”
Ben Allal: Pre-training on rewritten C4 web data outperforms raw C4
“They rewrite some samples from C four into Q and A into Wikipedia, and they find that doing this works better than training just on C four.”
Hugging Face Filtered 15T Token Dataset Down to 1.5T Educational Tokens
“And then we run this classifier on all of fine web, which is a 15 trillion tokens data set. And then we only keep the pages that have like a score that's higher than three. So for example, in our case, we went from 15 trillion tokens to just 1.5 trillion token…”
Ben Allal: FineWeb-Edu Outperforms All Other Public Web Datasets
“And as you can see here FineWebEDU outperforms all the other public web datasets by a larger margin on a couple of benchmarks.”
Ben Allal: Pooling multiple teacher models produces superior synthetic datasets
“Synthetic data, it doesn't have to come from a single model. And because we have so many good models now, you could like pull these models together and get like a dataset that's over really high quality and that's diverse and that's covers all your needs.”
Llama 3.2 1B matches Llama 2 13B performance on LMSYS Arena
“For example, Lama 3.21 B it matches Lama two 13 B from that was the release last year on the LMSS arena, which is basically the default go to leaderboard for evaluating models using human evaluation.”
Ben Allal: LLaMA 3 used 15x more pre-training tokens than original LLaMA
“LAMA was trained on one trillion tokens, but LAMA-III was trained on 15 trillion tokens.”
Meta research: Sub-1B language models benefit more from depth than width
“For example, they find that depth is more important than width, so it's more important to have models that have, like, more layers than just making them more wide. They also find that GQA helps, that tie-in the embedding helps, so I think it's a nice study ove…”
Ben Allal: Hugging Face SmolLM2-1.7B outperforms Llama 3.2 models
“So it's a series of three models, which are the best in class in each model size. For example, our 1.7 B model outperforms Lama one B and also .2.”
Ben Allal: Small models continue improving when trained on 11T tokens
“For example, smaller than one was trained only on one trillion tokens, but this model is trained on 11 trillion tokens. And we saw that the performance kept improving. The models didn't really plateau me training. Which I think is really interesting. It shows …”
Ben Allal: Small models make more sense than large models for text extraction
“So I think text extraction is like one use case where small models can be really performant, and it makes sense to use them instead of just using larger models.”
Ben Allal: AI industry will shift to fine-tuning over prompt engineering
“And I think we're going back to fine tuning where we realize these models are really cosplay. It's better to use just a small model. We try to specialize it. So I think it's a little bit of a cycle and we're going to start to see like more of fine tuning and l…”