Assertion Supported AI assessment confidence: 95% certainty 4/5 debate potential 2/5

Eysenbach: Scaling RL depth requires combining depth with residual connections

Benjamin Eysenbach · [NeurIPS Best Paper] 1000 Layer Networks for Self-Supervised RL — Kevin Wang et al, Princeton · Dec 31, 2025 · at 5:56

Princeton Professor Benjamin Eysenbach discusses the empirical findings behind their NeurIPS Best Paper on training 1000-layer reinforcement learning networks.

0:00 / 0:10exact quote · 10.4s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“And if we just made the depth bigger, it makes it worse. If we just add residual connections, it didn't make it better. And it was really this combination of factors that Kevin and Ishan figured out that really made this work.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

Scaling depth alone or adding residual connections alone does not improve RL performance, but combining them enables deep self-supervised RL to work.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

The published research by Kevin Wang, Ishaan Javali, Michał Bortkiewicz, Tomasz Trzcinski, and Benjamin Eysenbach ('1000 Layer Networks for Self-Supervised RL', presented at NeurIPS 2025) directly confirms all three legs of the claim: (1) Scaling depth alone in standard MLP architectures caused performance to collapse at moderate depths (e.g., depth 32) due to vanishing gradients and instability; (2) Adding residual connections alone to shallow networks resulted in marginal or no significant performance improvement; and (3) Combining deep architectures with residual connections (alongside LayerNorm and Swish activations) enabled scaling up to 1,024 layers, delivering 2x to 50x performance improvements across robotic manipulation and complex locomotion/maze navigation tasks.

NeurIPS 2025 paper '1000 Layer Networks for Self-Supervised RL' Section 4 / Figure 5 & 9 shows that scaling standard MLP depth without residual connections leads to complete performance collapse.
Ablations on shallow depths (2-4 layers) in Figure 5 confirm that adding residual connections alone yields only marginal differences.
Deep networks combining residual connections, LayerNorm, and Swish scaled up to 64, 256, and 1,024 layers achieved 2x-5x gains in manipulation and up to 50x gains in complex humanoid navigation tasks.

Checked against

[1] neurips.cc [2] github.io [3] openreview.net [4] princeton.edu [5] arxiv.org [6] arxiv.org [7] rewire.it [8] openreview.net

"1000 Layer Networks for Self-Supervised RL" "Figure 5" "without residual" OR "vanilla" OR "MLP" "1000 Layer Networks for Self-Supervised RL" "Figure 9" OR "Figure 5" "residual connections" "Benjamin Eysenbach" "residual connections" "depth" "self-supervised" NeurIPS "1000 Layer Networks for Self-Supervised RL" "residual connections" "without" OR "w/o" OR "ablation" "1000 Layer Networks for Self-Supervised RL" "Table 1" "ablation" "depth"

assessed as of 2026-09-02

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

Mentioned

Benjamin Eysenbach Ishan Gaur Kevin Wang residual connections

⚑ Spot an error? Report it

More from Benjamin Eysenbach

Insight

Eysenbach: 1,000-layer RL requires reward-free objectives, not just architectural tricks

“I think the main conclusion is that using big networks not only requires these architectural tricks, but also, as Kevin mentioned before, it requires using a different objective. This objective doesn't actually use rewards in it, and so there's another word in…”

Benjamin Eysenbach Dec 31, 2025 ▶ 8:08 [NeurIPS Best Paper] 1000 Layer Networks for Self-Supervised RL — Kevin Wang et al, Princeton

Assertion Supported

Eysenbach: Historically, 'deep' RL meant only two to four layers

“So it's like, probably my lab works on deep reinforcement learning, but historically deep meant like two or three or four layers.”

Benjamin Eysenbach Dec 31, 2025 ▶ 2:15 [NeurIPS Best Paper] 1000 Layer Networks for Self-Supervised RL — Kevin Wang et al, Princeton

Eysenbach: Scaling RL depth requires combining depth with residual connections

More from Benjamin Eysenbach

Turn any episode into a week of clips.