knowledge distillation
3 statements across 3 episodes · 2 bullish · 0 bearish · 3 people on the record · first statement Dec 24, 2024 by Loubna Ben Allal · across every show →
Everything said about knowledge distillation, oldest first
Dec 24, 2024 positive
Ben Allal: Distilling from large models avoids small-model self-training collapse
“I think if you do that approach, it's normal to observe this kind of behavior because the quality is going to be worse because the model is already small, and then if you train it just on these generations, you shouldn't expect it to become better. But what we…”
Mar 23, 2025 positive
Agarwal: Distilling a Large Model Outperforms Direct Training on the Same Data
“This is something that people have found again and again, that basically you can train a model on some data, or you can train a bigger model on that data and distill that model to another model, and that distill model is better.”
Feb 12, 2026
Jeff Dean: Distillation originated to compress 50-model ensembles into serviceable form
“Distillation was originally motivated because we were seeing that we had a very large image data set at the time, you know, three hundred million images that we could train on with, you know, I forget, like 20,000 categories or something, so much bigger than I…”