Loubna Ben Allal
Researcher, Hugging Face. On 1 show, 0 appearances, plus 1 compilation re-air not counted. The Shows tab opens the full record on each.
Loubna Ben Allal works on synthetic data and Smol Language Models at Hugging Face. She focuses on research involving synthetic data generation for pre-training, filtering, and model classification.
Everything Loubna Ben Allal said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Ben Allal: Recent web dumps improve model benchmarks despite synthetic data
“So what we did is we trained different models on these different dumps, and we then computed their performance on popular like NLP benchmarks, and then we computed the aggregated score. And surprisingly, you can see that the latest dumps are actually even bett…”
Ben Allal: Synthetic data may enrich the web rather than pollute it
“So personally, I wouldn't say the web is posted with synthetic data. Maybe it's even making it more rich.”
Ben Allal: Properly curated synthetic data prevents model collapse
“And I think there's a lot of concerns about model collapse, and I'm going to talk about that later, but we'll see that like, if we use synthetic data properly and we curate it carefully that shouldn't happen.”
Ben Allal: Hugging Face SmolLM2-1.7B outperforms Llama 3.2 models
“So it's a series of three models, which are the best in class in each model size. For example, our 1.7 B model outperforms Lama one B and also .2.”
Ben Allal: Small models continue improving when trained on 11T tokens
“For example, smaller than one was trained only on one trillion tokens, but this model is trained on 11 trillion tokens. And we saw that the performance kept improving. The models didn't really plateau me training. Which I think is really interesting. It shows …”
Ben Allal: AI industry will shift to fine-tuning over prompt engineering
“And I think we're going back to fine tuning where we realize these models are really cosplay. It's better to use just a small model. We try to specialize it. So I think it's a little bit of a cycle and we're going to start to see like more of fine tuning and l…”
Ben Allal: LLMs can be trained with entirely synthetic pipelines
“Today you can train an LLM with like an entirely synthetic pipeline. For example, you can use our Cosmopedia data sets and you can train a one B model on like a hundred and fifty billion tokens. Those are a hundred percent synthetic, and those are also of good…”
Ben Allal: Distilling from large models avoids small-model self-training collapse
“I think if you do that approach, it's normal to observe this kind of behavior because the quality is going to be worse because the model is already small, and then if you train it just on these generations, you shouldn't expect it to become better. But what we…”
Ben Allal: Prompt diversity is essential for scaling synthetic training data
“The key ingredient to getting a good data set that is synthetic is trying as much as possible to keep it diverse, because if you just throw the same prompts as your model, like generate, like, a textbook about linear algebra, and even if you change the tempera…”
Synthetic college text boosts MMLU; middle school text boosts OpenBookQA
“College textbooks are really good for MLU or middle school textbooks are good for benchmarks like open book, UA and Pico.”
Ben Allal: Small models can generate synthetic data by rephrasing web pages
“The interesting thing in this approach is that you can use a model that is small Because it doesn't, rewriting doesn't require knowledge. It's just rewriting a page into a different style. So the model doesn't need to have like knowledge that is like extensive…”
Ben Allal: Pre-training on rewritten C4 web data outperforms raw C4
“They rewrite some samples from C four into Q and A into Wikipedia, and they find that doing this works better than training just on C four.”
Ben Allal: FineWeb-Edu Outperforms All Other Public Web Datasets
“And as you can see here FineWebEDU outperforms all the other public web datasets by a larger margin on a couple of benchmarks.”
Ben Allal: Pooling multiple teacher models produces superior synthetic datasets
“Synthetic data, it doesn't have to come from a single model. And because we have so many good models now, you could like pull these models together and get like a dataset that's over really high quality and that's diverse and that's covers all your needs.”
Llama 3.2 1B matches Llama 2 13B performance on LMSYS Arena
“For example, Lama 3.21 B it matches Lama two 13 B from that was the release last year on the LMSS arena, which is basically the default go to leaderboard for evaluating models using human evaluation.”
Meta research: Sub-1B language models benefit more from depth than width
“For example, they find that depth is more important than width, so it's more important to have models that have, like, more layers than just making them more wide. They also find that GQA helps, that tie-in the embedding helps, so I think it's a nice study ove…”
Ben Allal: Small models make more sense than large models for text extraction
“So I think text extraction is like one use case where small models can be really performant, and it makes sense to use them instead of just using larger models.”
Hugging Face finds LLM proxy words jumped in Common Crawl after ChatGPT
“For example, here we measured like these words ratio in different dumps of common crawl, and we can see that like the ratio really increased after chat GPT's release.”
Ben Allal: NVIDIA generated 1.9 trillion synthetic tokens for Nemotron-CC
“This is a recent paper from NVIDIA, Mnemotron CC. They took things a bit further and they generated not a few billion tokens, but 1.9 trillion tokens, which is huge.”
Hugging Face Filtered 15T Token Dataset Down to 1.5T Educational Tokens
“And then we run this classifier on all of fine web, which is a 15 trillion tokens data set. And then we only keep the pages that have like a score that's higher than three. So for example, in our case, we went from 15 trillion tokens to just 1.5 trillion token…”
The other half of the tape: Loubna Ben Allal's own voice is left out of every number here. Other people bring the name up 3 times in 2 episodes across the shows. every mention, with the transcript →
Who brings them up most Shawn Wang 2Elie Bakouch 1
Every mention by year
tap a year for its mentions
Latent Space 3
One line per show, most statements first. The link opens Loubna's full record on that show: the calibration, argument clarity, speaking style and every statement made there.
| Show | Role there | Eps | Statements | Record | |
|---|---|---|---|---|---|
| LATENT SPACELEDGER | Researcher, Hugging Face | 0 +1 | 21 | — | full record on Latent Space → |