distillation loss
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Mar 23, 2025 by Rishabh Agarwal · across every show →
Everything said about distillation loss, oldest first
Mar 23, 2025 neutral
Agarwal: Standard distillation KL loss places mass where teacher has none
“The typically what we use is this mode covering KL, the one on the left. That is the standard distillation loss that everyone uses. But you can see the behavior. And you can already see what's weird about it. It's putting a lot of mass on places where there's …”