Elie Bakouch
Pre-training Lead, Hugging Face. On 1 show, 1 appearance. The Shows tab opens the full record on each.
Elie Bakouch leads pre-training efforts at Hugging Face and is an architect behind SmolLM. He specializes in large language model architecture design, training stability, and dataset optimization.
1shows
1appearances
13statements
4resolved
4supported
0contradicted
100%fully supported
Everything Elie Bakouch said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Bakouch: Novel optimizer speedups are exaggerated due to undertuned AdamW baselines
“And what they find is that the speed up is greatly, greatly exaggerated. And mostly because often people like undertone the Adam W baseline.”
Bakouch primarily uses closed AI models for deep research out of convenience
“Yeah, I think even I, to be honest, I'm mainly using a closed model because it's easier, basically. It's like the same interface when I'm like, for example, researching very deep on papers.”
Bakouch: DeepSeek and Qwen do not release all their ablation data
“We want to train our MOE because it's fun and everyone is doing that. And also I think there is a lot of different direction. And it's always good in terms of science. To, because basically the coin tree or even deep seek, they don't release, release all the a…”
Bakouch: Megatron's local batching hindered MoE expert specialization
“And they find that, for example, in the Megatron item code base this wasn't the case. This was done as the local batch I think. So it basically means that everyone that is using Megatron at the time of the paper I mean, it wasn't even a paper. It was just a bl…”
Bakouch: Open-source AI is not really doing distillation yet
“Actually, distillation is like quite a big thing that people in the open are not really
Really doing yet, I think”
Bakouch: SmolLM 2 scored random on MMLU until 6.5 trillion tokens
“And this is basically until like 6.5 trillion of tokens, which is a lot, to be honest. Until this amount of token, the MMLU in the QA format, meaning that the model have to select which answer is, the model have to output, for example, the right answer is A, o…”
Bakouch: Multi-token prediction is particularly effective for coding models
“Multi-token prediction, which is very good for example, for coding.”
Bakouch: DeepSeek-V3 uses the same Adam optimizer parameters as Llama 2
“And for example, a good a good way to view that is that DeepSeq rig three is still using the same Adam parameter than Lama two.”
Bakouch: Optimizer ablations must wait for full learning rate decay
“Especially when doing optimizer ablation, you never want to look at early curve, and you want to, like, wait for the long rate to have fully decay, and, like, do anything to be to zero, or whatever value.”
Bakouch: Hugging Face plans to train an MoE model soon
“For example, we tried we are training MOE currently at TargetFace. I mean, we'll train soon. We start the training soon. And we tried with Megatron and we benchmarked, like, for example, the Mistral architecture with the Queen's three this one.”
Bakouch: Hugging Face research showed diminishing returns after 3 data epochs
“There is this paper from actually from people at HuginFace at the time that is saying that you basically can repeat your data Like, up to three epochs before seeing diminishing gain before seeing diminishing return.”
Hugging Face's pre-training research team averages 30 people
“We also have this team working on pre-training and training models such as small LM. And we are basically a very small team of We have 30 people in average”
Bakouch: Hugging Face introduces code and math in 1T+ token phase
“We are basically introducing higher quality data like I don't know where it is, but if you look at the different phase, we are basically adding more code and math through the train and less web data. But we're only doing it like one point trillion token.”
One line per show, most statements first. The link opens Elie's full record on that show: the calibration, argument clarity, speaking style and every statement made there.
| Show | Role there | Eps | Statements | Record | |
|---|---|---|---|---|---|
 LATENT SPACELEDGER | Pre-training Lead, Hugging Face | 1 | 13 | 100% 4/4 | full record on Latent Space → |