Insight certainty 4/5 debate potential 3/5

Eysenbach: 1,000-layer RL requires reward-free objectives, not just architectural tricks

Benjamin Eysenbach · [NeurIPS Best Paper] 1000 Layer Networks for Self-Supervised RL — Kevin Wang et al, Princeton · Dec 31, 2025 · at 8:08

Benjamin Eysenbach, Princeton professor and co-author of the NeurIPS Best Paper on 1,000-layer RL networks, clarifies the core requirement for scaling deep models in RL.

0:00 / 0:21exact quote · 21.8s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“I think the main conclusion is that using big networks not only requires these architectural tricks, but also, as Kevin mentioned before, it requires using a different objective. This objective doesn't actually use rewards in it, and so there's another word in the title reinforcement learning that also might be a little bit of a misnomer because we aren't directly trying to maximize rewards.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

Successfully scaling deep neural networks to 1,000 layers in reinforcement learning requires replacing traditional reward-maximization objectives with a reward-free objective, rather than simply plugging deep architectures into algorithms like PPO or SAC.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Mentioned

Benjamin Eysenbach PPO reinforcement learning SAC

⚑ Spot an error? Report it

More from Benjamin Eysenbach

Assertion Supported

Eysenbach: Scaling RL depth requires combining depth with residual connections

“And if we just made the depth bigger, it makes it worse. If we just add residual connections, it didn't make it better. And it was really this combination of factors that Kevin and Ishan figured out that really made this work.”

Benjamin Eysenbach Dec 31, 2025 ▶ 5:56 [NeurIPS Best Paper] 1000 Layer Networks for Self-Supervised RL — Kevin Wang et al, Princeton

Assertion Supported

Eysenbach: Historically, 'deep' RL meant only two to four layers

“So it's like, probably my lab works on deep reinforcement learning, but historically deep meant like two or three or four layers.”

Benjamin Eysenbach Dec 31, 2025 ▶ 2:15 [NeurIPS Best Paper] 1000 Layer Networks for Self-Supervised RL — Kevin Wang et al, Princeton

Eysenbach: 1,000-layer RL requires reward-free objectives, not just architectural tricks

More from Benjamin Eysenbach

Turn any episode into a week of clips.