recurrent neural networks

also referred to as: rnns

3 statements across 3 episodes · 2 bullish · 0 bearish · 3 people on the record · first statement Aug 3, 2023 by Tri Dao · across every show →

Everything said about recurrent neural networks, oldest first

Aug 3, 2023 bullish

Prediction Not checkable as stated

RNNs will outperform Transformers in batch generation and long sequences

“I am personally bullish on, on, on RNNs. I think RNNs they don't, they essentially summarize the past into a state vector. They have fixed size, so the size doesn't grow with the history. So that means that you don't need as much memory to keep around all the …”

Tri Dao Aug 3, 2023 ▶ 49:14 FlashAttention-2: Making Transformers 800% faster AND exact

Aug 31, 2023 bullish

Assertion Supported

RWKV Trains in Parallel Across GPUs, Unlike Traditional RNNs

“And in practice, once you start cascading there, you just saturate the GPU, and that's how it starts being paralysable trained. You no longer need to train in slices like traditional RNNs.”

Eugene Cheah Aug 31, 2023 ▶ 58:45 RWKV: Reinventing RNNs for the Transformer Era

Jul 2, 2025 neutral

What-if

Morris: ChatGPT could likely have been built using RNNs instead of Transformers

“And I think like, we honestly probably could have gotten this with RNNs. I know like the scaling laws paper shows that RNNs have worse curves for scaling, but probably people would have been like, I bet you could have built chat GPT with a very sophisticated R…”

Jack Morris Jul 2, 2025 ▶ 1:10:28 Information Theory for Language Models: Jack Morris

Turn any episode into a week of clips.

This entire site, over 200 episodes transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

recurrent neural networks

Everything said about recurrent neural networks, oldest first

Turn any episode into a week of clips.