LSTM

“The state of the art at this time was LSTMs, Recurring Neural Networks, which you could not launch in production at all because they're too slow and take a fair bit of time to process as documents scale.”

“We come from a world where we had recurrent neural networks and LSTMs that actually had infinite memory, although it was not very capable, right? You, the models in, in practice, they never remember more than a few hundred words or so.”

“When people talk about the scaling loss in neural networks, the scaling laws are actually a to a large extent of a property of the transformer. Before the transformer, people were playing with LSTMs and stacking them, etc. You don't actually get like clean sca…”