Ben Allal: AI industry will shift to fine-tuning over prompt engineering
Loubna Ben Allal · Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024] · Dec 24, 2024 · at 27:47
This episode carries Loubna Ben Allal's own address, with nobody on the show putting questions to them. It still counts as said, and it is kept out of every score on their page.
Loubna Ben Allal of Hugging Face gives her outlook on AI workflows and model optimization at NeurIPS 2024.
0:00 / 0:13exact quote · 13.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And I think we're going back to fine tuning where we realize these models are really cosplay. It's better to use just a small model. We try to specialize it. So I think it's a little bit of a cycle and we're going to start to see like more of fine tuning and less of just like a prompt engineering the models.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Loubna Ben Allal
Ben Allal: Recent web dumps improve model benchmarks despite synthetic data
“So what we did is we trained different models on these different dumps, and we then computed their performance on popular like NLP benchmarks, and then we computed the aggregated score. And surprisingly, you can see that the latest dumps are actually even bett…”
Ben Allal: Synthetic data may enrich the web rather than pollute it
“So personally, I wouldn't say the web is posted with synthetic data. Maybe it's even making it more rich.”
Ben Allal: Properly curated synthetic data prevents model collapse
“And I think there's a lot of concerns about model collapse, and I'm going to talk about that later, but we'll see that like, if we use synthetic data properly and we curate it carefully that shouldn't happen.”
Ben Allal: Hugging Face SmolLM2-1.7B outperforms Llama 3.2 models
“So it's a series of three models, which are the best in class in each model size. For example, our 1.7 B model outperforms Lama one B and also .2.”
Ben Allal: Small models continue improving when trained on 11T tokens
“For example, smaller than one was trained only on one trillion tokens, but this model is trained on 11 trillion tokens. And we saw that the performance kept improving. The models didn't really plateau me training. Which I think is really interesting. It shows …”
Ben Allal: LLMs can be trained with entirely synthetic pipelines
“Today you can train an LLM with like an entirely synthetic pipeline. For example, you can use our Cosmopedia data sets and you can train a one B model on like a hundred and fifty billion tokens. Those are a hundred percent synthetic, and those are also of good…”