Transformer Architecture
topic on 9 shows · 20 statements across 20 episodes
More or Less 
Latent Space 
Lenny's Podcast 
No Priors 
WTF is with Nikhil Kamath 
the MAD Podcast 
Big Technology 
All-In 
20VC
20 statements about Transformer Architecture, every show
Anandkumar: Standard Transformers cannot scale to 5-trillion context lengths for physics
“On the other hand, if you think about using transformer architectures that have worked so well for language, that just wouldn't be able to support a five trillion context length. No matter all the compute in the world is thrown at it. So that kind of quadratic…”
Midha: Anthropic's velocity came from standardizing on the transformer architecture
“Like, one of the reasons Anthropic has had extraordinary sort of velocity is because they picked the transform architecture and said, this is simple, let's double down on it, right? And now, luckily, there's enough investment going into space that we can affor…”
Ries: Corporations are the oldest form of artificial intelligence
“Corporations, organizations are the oldest form of artificial intelligence on the planet. They are an example of this emergent intelligence, the same scientific principle that makes the transformer architecture work and appear intelligent. That same principle …”
AI capabilities would have been achieved even without inventing transformers
“I think if we hadn't invented the transformer, we would have gotten there with whatever LSTM you know, state space model, whatever, anything else people were developing, we would have gotten there.”
Vision Transformers benefited heavily from software infrastructure built for language models
“There was also benefit of doing that simply because the rest of the, that the machine learning field, which was working on, on, on language, they were using this, like architecture. So they were building infra for it, making it faster. And, you know, like the,…”
Transformer remains state of the art for LLM performance
“I would say right now, yes, because it's still the state of the art. So there is nothing really better in terms of state of the art performance, getting better quality results.”
Feldman: Transformer architecture has several more years of viability
“And finally, I think the transformer, the current architecture has a way still to run. I think we will see that for several years more.”
Frosst: Core transformer architecture has remained largely unchanged for a decade
“The models themselves, like transformer architecture, which is the original model that, yeah, that was introduced in Hasn't changed very much, right? Like all, the whole industry is still using transformers. We've changed the way we train them, but the model a…”
Gomez: Google quickly integrated Transformer architecture into Search and Translate
“No, they jumped all over it. So it went to production inside of search, inside of translate, like the existing product suite. And so they, to say that they didn't adopt the transformer architecture would not be correct.”
Shankar: LLM failure modes and evaluation techniques have largely stabilized
“I think techniques have stabilized. I think the kinds of failure modes of LLMs, I mean, they're still there, but it's not like changing every single day. We know that LLMs are bad at certain things. We know a little bit more about say limitations of the transf…”
Kiela: Attention mechanism, not Transformers, was the real AI breakthrough
“So I would say, and maybe I'm biased because one of my best friends is, is on the original attention paper, but that was the real breakthrough. It's just like figuring out that you have this attention mechanism that actually allows you to yeah, to do a much be…”
LLMs primarily perform data retrieval and possess very little actual reasoning
“If it's text, they will regurgitate solutions to puzzles. They will, you know, give you answers to questions you may have. It's mostly retrieval. There's a very tiny bit of reasoning, but really not much and that's an important limitation.”
Polu: OpenAI Believed in Transformer Scaling Pre-Kaplan Paper
“Before that, there really was a strong belief in, in scale. I think it was just the belief that the transformer was a generic enough architecture that you could learn anything, and that this was just a question of scaling.”
Dave Morin: AI progress will probably plateau longer than people think
“A simple way to think of it is like, we're probably gonna be kind of where we are right now for a lot longer than people think. If the current architecture and the current energy and the current capital needs are the gating factors.”
Sutskever: Transformers can reach AGI; alternatives only offer compute efficiency
“So it's better to think about it in terms of compute efficiency rather than in terms of, can it get there at all? I think at this point, the answer is obviously yes.”
Nemade: Google teams only noticed Transformer paper after citations grew
“I think once it started garnering a lot of citations over the months and quarters, that's when people started paying attention, even inside of Google that, Hey, this transformer architecture seems amazing.”
Gil: GPU and Transformer lock-in is stronger than 1990s Wintel
“And so this is, I feel like, a stronger version of that in some sense, where you have the underlying compute architecture and the most important model reinforcing each other in a way that kind of locks both of them in.”
Uszkoreit: Community optimism drove Transformer adoption and success
“The other main contributor, I think, to the success of this architecture was optimism and hope. So suddenly you were in a situation where, for whatever reason, a bunch of things that people tried with this started to work, and then more started to work, and th…”
LeCun: Transformer scaling with self-supervised learning surprised AI researchers
“The fact that self-supervised learning methods applied to transformer architectures Work amazingly well, and they work, you know, way beyond what we could have expected. The fact that we can do basically train systems to understand language, translate language…”