GRPO

“Like what we, what OpenAI is doing is not exactly GRPO. It is slightly different in many different ways, but like some parts are definitely similar.”

“In reality, it seems like most people are using something like an evolved version of GRPO, which is a bit simpler than PPO.”

“It's similar to, you know, optimizers with Adam. So Adam is, I mean, right now there's Adam W. There was SGD and all the other RMS prop and how they were called. And they kind of all converge to Adam W by adding more and more tricks. And I think that's the sam…”

“Vanilla GRP or the original algorithm, it is pretty flaky. Like where it is, you have to babysit it. Over the course of the year, many people had these tips and tricks where some people were saying, remove the KL divergence term. Like if you just drop it for m…”