Transformers Beat Recurrent Models by Processing Whole Sequences at Once
Noam Shazeer · No Priors Ep. 12 | With Noam Shazeer · Apr 25, 2023 · at 4:50
Noam Shazeer, co-author of the Transformer paper and co-founder of Character.AI, explains the core computational advantage of Transformers over sequential recurrent neural networks.
0:00 / 0:37exact quote · 37.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The magic of transformer kind of like convolutions is that you get to process the whole sequence at once. I mean, it still talks, you know, it's still a function of like the, you know, the predictions for the later words are dependent on what the earlier words are, but it happens in like a constant number of steps where you get to take advantage of like this parallelism. If you can look at like the whole thing at once and like, that's what modern hardware is good at is parallelism. And now you can use the length of the sequence as your parallelism and everything works, you know, super well.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Noam Shazeer
Shazeer: Most Core AI Intelligence Will Come From Text Models
“But I think that a lot of the core intelligence is going to come from a From these text models.”
Shazeer Calls Next-Token Prediction an AI-Complete Problem
“And like, the problem is super simple to define. It's just like predict the next word, the fat cat sat on the, you know, like, okay, well, you know, what comes next? Like it's extremely easy to define. And if you can do a great job of it, like, you know, then …”
Core AI Intelligence Will Primarily Emerge From Text Models
“But I think that a lot of the core intelligence is going to come from these text models.”
AI Scaling Has Not Hit a Wall and Will Keep Improving
“So, and at the same time, like, I don't think anyone's seen a wall in terms of how good this stuff is. So I think it will just, it's just gonna keep getting better. I don't know what stops it.”
AI Models Can Generate Synthetic Data to Avoid Looming Shortages
“I think I'm not that worried about Coming up with data and I feel like we could probably like just generate some more with the AI.”
AI Assistants With a Single Neutral Corporate Persona End Up Boring
“Like if you're like such a public, you know, trying to present like one public persona that everyone likes, you're going to end up just being boring essentially. And people just don't want boring. You know, people want like the, you know want to interact with …”