Knowledge Distillation
topic on 1 show · 3 statements across 3 episodes
3 statements about Knowledge Distillation, every show
Jeff Dean: Distillation originated to compress 50-model ensembles into serviceable form
“Distillation was originally motivated because we were seeing that we had a very large image data set at the time, you know, three hundred million images that we could train on with, you know, I forget, like 20,000 categories or something, so much bigger than I…”
Agarwal: Distilling a Large Model Outperforms Direct Training on the Same Data
“This is something that people have found again and again, that basically you can train a model on some data, or you can train a bigger model on that data and distill that model to another model, and that distill model is better.”
Ben Allal: Distilling from large models avoids small-model self-training collapse
“I think if you do that approach, it's normal to observe this kind of behavior because the quality is going to be worse because the model is already small, and then if you train it just on these generations, you shouldn't expect it to become better. But what we…”