KL divergence
3 statements across 1 episodes · 2 bullish · 0 bearish · 1 people on the record · first statement Mar 23, 2025 by Rishabh Agarwal · across every show →
Everything said about KL divergence, oldest first
Mar 23, 2025 positive
Agarwal: Inverting KL Divergence Direction Derives On-Policy Distillation
“All you need to do, at least in math, if you just flip the direction of your KL divergence, so earlier we were minimizing KL between a teacher and student model, if we just swap that order, because KL is not symmetric, you will actually get this kind of distil…”
Mar 23, 2025 positive
Agarwal: 50/50 mix of KL divergences usually works when goals are unclear
“The general recommendation I would give people is that maybe use a mixture of half and half. Like that's what some people have used, right? That's like saying, yeah, basically saying, I don't know what I want. I just want something to work well enough. I'll ju…”
Mar 23, 2025 neutral
Agarwal: Distillation KL direction dictates trade-off between diversity and performance
“There's a trade-off between diversity and performance. So on the y-axis, I'm showing performance. On the x-axis, I'm showing similarity or basically how, like one minus diversity. So more similar things are less diverse. And you can see, depending on the diver…”