Recurrent Neural Networks
topic on 3 shows · 10 statements across 8 episodes
the Y Combinator Startup Podcast 
Latent Space 
No Priors
10 statements about Recurrent Neural Networks, every show
Chaubard: Standard LLMs sacrifice latent time compression unlike RNNs
“What you actually paid for that you have to give up is this latent reasoning thing and this compression in the time direction. There is no compression in LMs. Every single decode that I do, I still have to retain the entire, you know, Shakespeare novel just to…”
Chaubard: Hierarchical Reasoning Models Offer Little Novelty Over Standard RNNs
“The, this is directly in the lineage of RNNs. There's not that much novel from, like, the RNN standpoint at least in my opinion.”
Morris: ChatGPT could likely have been built using RNNs instead of Transformers
“And I think like, we honestly probably could have gotten this with RNNs. I know like the scaling laws paper shows that RNNs have worse curves for scaling, but probably people would have been like, I bet you could have built chat GPT with a very sophisticated R…”
Vinyals: RNNs and LSTMs Never Remembered Beyond a Few Hundred Words
“We come from a world where we had recurrent neural networks and LSTMs that actually had infinite memory, although it was not very capable, right? You, the models in, in practice, they never remember more than a few hundred words or so.”
RWKV Trains in Parallel Across GPUs, Unlike Traditional RNNs
“And in practice, once you start cascading there, you just saturate the GPU, and that's how it starts being paralysable trained. You no longer need to train in slices like traditional RNNs.”
RNNs will outperform Transformers in batch generation and long sequences
“I am personally bullish on, on, on RNNs. I think RNNs they don't, they essentially summarize the past into a state vector. They have fixed size, so the size doesn't grow with the history. So that means that you don't need as much memory to keep around all the …”
Transformers Beat Recurrent Models by Processing Whole Sequences at Once
“The magic of transformer kind of like convolutions is that you get to process the whole sequence at once. I mean, it still talks, you know, it's still a function of like the, you know, the predictions for the later words are dependent on what the earlier words…”
Sidor: Rigorous baseline optimization will advance AI more than complex architectures
“And you know, it's not the kind of sexy research that people want to see, where you have like some hierarchy of big RNN, but it actually, this kind of research, I think at this point will advance field the most.”
Eck: Google Magenta's early music generation models were primitive RNNs
“We put out some models that were, you know, by any reasonable account primitive, I mean, kind of very simple recurrent neural networks that that generate MIDI from MIDI”