Ben Allal: Distilling from large models avoids small-model self-training collapse
Loubna Ben Allal · Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024] · Dec 24, 2024 · at 4:56
This episode carries Loubna Ben Allal's own address, with nobody on the show putting questions to them. It still counts as said, and it is kept out of every score on their page.
Loubna Ben Allal explains why synthetic data in practice avoids the model collapse dynamics described in academic papers.
0:00 / 0:21exact quote · 21.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think if you do that approach, it's normal to observe this kind of behavior because the quality is going to be worse because the model is already small, and then if you train it just on these generations, you shouldn't expect it to become better. But what we're really doing here is that we take a model that is very large, And we try to distill its knowledge into a model that is smaller, and in this way you can expect to get, like, a better performance for your small model.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Loubna Ben Allal
Ben Allal: Recent web dumps improve model benchmarks despite synthetic data
“So what we did is we trained different models on these different dumps, and we then computed their performance on popular like NLP benchmarks, and then we computed the aggregated score. And surprisingly, you can see that the latest dumps are actually even bett…”
Ben Allal: Synthetic data may enrich the web rather than pollute it
“So personally, I wouldn't say the web is posted with synthetic data. Maybe it's even making it more rich.”
Ben Allal: Properly curated synthetic data prevents model collapse
“And I think there's a lot of concerns about model collapse, and I'm going to talk about that later, but we'll see that like, if we use synthetic data properly and we curate it carefully that shouldn't happen.”
Ben Allal: Hugging Face SmolLM2-1.7B outperforms Llama 3.2 models
“So it's a series of three models, which are the best in class in each model size. For example, our 1.7 B model outperforms Lama one B and also .2.”
Ben Allal: Small models continue improving when trained on 11T tokens
“For example, smaller than one was trained only on one trillion tokens, but this model is trained on 11 trillion tokens. And we saw that the performance kept improving. The models didn't really plateau me training. Which I think is really interesting. It shows …”
Ben Allal: AI industry will shift to fine-tuning over prompt engineering
“And I think we're going back to fine tuning where we realize these models are really cosplay. It's better to use just a small model. We try to specialize it. So I think it's a little bit of a cycle and we're going to start to see like more of fine tuning and l…”