Oct 20, 2025 · 1h 3m · latent-space
⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space Podcast, Hugging Face researcher Elie Bakouch delivers a comprehensive masterclass on open-source LLM pretraining. He covers end-to-end dataset curation, advanced optimizers like Muon, Mixture of Experts (MoE) routing architectures, and the deployment of compact, on-device language models.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 22.6% of the talking time here. How this is scored →
The hosts as informed peer 5.8 Guest teaching 6.3 Guest disagreement 1.7 The hosts pushing back 2.5
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 16:04 Elie dismisses hyped optimizer speedups
Elie rejects claims of massive speedups in recent optimizer papers, noting that authors frequently undertune the AdamW baseline to exaggerate their gains.
Hardest push from the hosts ▶ 46:20 Swyx pushes back against over-sanitized datasetsSwyx challenges the trend of web rephrasing, arguing that scrubbing all typos and grammatical imperfections leaves models unprepared for real user prompts.
Biggest teaching moment ▶ 23:15 Elie details Megatron's micro-batch load balancing issueElie demonstrates how Megatron's micro-batch routing calculation prevented expert domain specialization, educating the hosts on a crucial distributed MoE training detail.
The host holds their own ▶ 1:00:10 Alessio on the broken economics of hosting sub-1B modelsAlessio explains the economic and infrastructure bottleneck of serving sub-billion parameter models under existing hourly Hugging Face API pricing.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| A Unified Framework for LLM Pretraining | 6 | 6 | 1 | 2 | Alessio and Swyx ask targeted questions about pretraining stages and model scaling frontiers, while Elie outlines his five-pillar framework for LLM training and points out anomalies like DeepSeek using Llama 2 Adam parameters. | |
| Deep Dive into Optimizers: AdamW, Muon, and Stability | 5 | 7 | 2 | 2 | Swyx probes into the mechanics of newer optimizers like Muon and Defazio's schedule-free approach, prompting Elie to explain Newton-Schulz orthogonalization and debunk exaggerated speedup claims in recent papers. | |
| Mixture of Experts (MoE) Architectures and Routing Dynamics | 6 | 8 | 2 | 3 | The hosts discuss MoE sparsity, distillation, and routing mechanics. Elie dives deep into global vs. local batch load-balancing statistics in Megatron and explains shared vs. zero-communication experts in cutting-edge models. | |
| Dataset Engineering: Rephrasing, Formats, and Epoch Scaling | 7 | 6 | 2 | 4 | Swyx actively pushes on the pitfalls of synthetic rephrasing making datasets 'too clean' to handle typos and user errors. Elie agrees with the premise and shares empirical findings on conversational format impact and epoch repetition limits. | |
| Open Science with SmolLM, Open-Source Tooling, and Training Realities | 5 | 6 | 1 | 1 | The conversation turns to open science practices, Datatrove, LightEval, and training war stories, with Elie showing loss curve spikes from bugged optimizer ablations. | |
| On-Device AI, Small Model Ecosystem, Modular, and Conclusion | 6 | 5 | 2 | 3 | Alessio and Swyx probe into deployment economics and whether Elie has evaluated Modular's stack. Elie candidly admits he hasn't looked closely into Mojo/Modular because changing pretraining codebases is high friction. |
Statements from this episode (16)
Swix: SmolLM 3 is the best open-source AI paper in two years
“I think that, you know, honestly, SmallLM three, one of the single best open source AI research papers I've read, I think probably in like a year, maybe two years.”
Hugging Face's pre-training research team averages 30 people
“We also have this team working on pre-training and training models such as small LM. And we are basically a very small team of We have 30 people in average”
Bakouch: Multi-token prediction is particularly effective for coding models
“Multi-token prediction, which is very good for example, for coding.”
Bakouch: DeepSeek-V3 uses the same Adam optimizer parameters as Llama 2
“And for example, a good a good way to view that is that DeepSeq rig three is still using the same Adam parameter than Lama two.”
Swix: Frontier model sizes have likely peaked around 2 trillion parameters
“I wonder if we've hit the peak big model craze because now I do expect, you know, 10 trillion model releases, you know, 100 trillion model releases. Probably not. I think we might have peaked at two.”
Bakouch: Novel optimizer speedups are exaggerated due to undertuned AdamW baselines
“And what they find is that the speed up is greatly, greatly exaggerated. And mostly because often people like undertone the Adam W baseline.”
Bakouch: Optimizer ablations must wait for full learning rate decay
“Especially when doing optimizer ablation, you never want to look at early curve, and you want to, like, wait for the long rate to have fully decay, and, like, do anything to be to zero, or whatever value.”
Bakouch: Megatron's local batching hindered MoE expert specialization
“And they find that, for example, in the Megatron item code base this wasn't the case. This was done as the local batch I think. So it basically means that everyone that is using Megatron at the time of the paper I mean, it wasn't even a paper. It was just a bl…”
Bakouch: Hugging Face plans to train an MoE model soon
“For example, we tried we are training MOE currently at TargetFace. I mean, we'll train soon. We start the training soon. And we tried with Megatron and we benchmarked, like, for example, the Mistral architecture with the Queen's three this one.”
Swix: Every frontier lab now distills dense models into MoEs
“I think like, I think this is the pattern for every frontier lab now.”
Bakouch: Open-source AI is not really doing distillation yet
“Actually, distillation is like quite a big thing that people in the open are not really
Really doing yet, I think”
Bakouch: SmolLM 2 scored random on MMLU until 6.5 trillion tokens
“And this is basically until like 6.5 trillion of tokens, which is a lot, to be honest. Until this amount of token, the MMLU in the QA format, meaning that the model have to select which answer is, the model have to output, for example, the right answer is A, o…”
Bakouch: Hugging Face research showed diminishing returns after 3 data epochs
“There is this paper from actually from people at HuginFace at the time that is saying that you basically can repeat your data Like, up to three epochs before seeing diminishing gain before seeing diminishing return.”
Bakouch: Hugging Face introduces code and math in 1T+ token phase
“We are basically introducing higher quality data like I don't know where it is, but if you look at the different phase, we are basically adding more code and math through the train and less web data. But we're only doing it like one point trillion token.”
Bakouch primarily uses closed AI models for deep research out of convenience
“Yeah, I think even I, to be honest, I'm mainly using a closed model because it's easier, basically. It's like the same interface when I'm like, for example, researching very deep on papers.”
Bakouch: DeepSeek and Qwen do not release all their ablation data
“We want to train our MOE because it's fun and everyone is doing that. And also I think there is a lot of different direction. And it's always good in terms of science. To, because basically the coin tree or even deep seek, they don't release, release all the a…”