Distillation Loss
topic on 1 show · 1 statements across 1 episodes
1 statements about Distillation Loss, every show
Agarwal: Standard distillation KL loss places mass where teacher has none
“The typically what we use is this mode covering KL, the one on the left. That is the standard distillation loss that everyone uses. But you can see the behavior. And you can already see what's weird about it. It's putting a lot of mass on places where there's …”