on-policy distillation

“All you need to do, at least in math, if you just flip the direction of your KL divergence, so earlier we were minimizing KL between a teacher and student model, if we just swap that order, because KL is not symmetric, you will actually get this kind of distil…”

“So this kind of thing was used in Jemma DuPo's training. It's mentioned briefly, like, there's no details there, but it was used there.”

“The other thing is in the RL literature, there are, like, results which show that actually this kind of distillation is much more optimal for agentic tasks or really long horizon tasks.”