PPO
4 statements across 3 episodes · 3 bullish · 1 bearish · 3 people on the record · first statement Jan 11, 2024 by Nathan Lambert · said 1 times in 1 episodes since 2025 · across every show →
Mentions by year
brought up most by Benjamin Eysenbach (1)
tap a year for its mentions
Everything said about PPO, oldest first
Jan 11, 2024 negative
Jan 11, 2024 positive
Lambert: Meta Used Rejection Sampling to Bootstrapping Llama 2 RLHF
“Llama started their RLHF process with this to get some signal out of preference data. That preference data went into a reward model, and then the reward model did a good enough ranking that it was, like, essentially superpowered instruction tuning based on rew…”
Oct 16, 2025 positive
Corbitt: PPO enables training purely on real production traces without simulation
“And PPO, now in practice, a lot of times when you're training with PPO, you also will use an environment like that because it lets you do a bunch of runs and be more data efficient. But at least in principle, you have the option with PPO, you can actually, lik…”
Dec 31, 2025 positive
Eysenbach: 1,000-layer RL requires reward-free objectives, not just architectural tricks
“I think the main conclusion is that using big networks not only requires these architectural tricks, but also, as Kevin mentioned before, it requires using a different objective. This objective doesn't actually use rewards in it, and so there's another word in…”