Transformer
also referred to as: transformers
28 statements across 20 episodes · 17 bullish · 5 bearish · 19 people on the record · first statement Apr 25, 2023 by Dr. Percy Liang · said 13 times in 6 episodes since 2023 · across every show →
Mentions by year
brought up most by Sarah Guo (4), Jensen Huang (4), Elad Gil (2), Liam Fedus (1), Illia Polosukhin (1), Andrej Karpathy (1)
tap a year for its mentions
2026 1 mention in 1 episode
2024 1 mention in 1 episode
Everything said about Transformer, oldest first
Apr 25, 2023 positive
Liang: Stanford researchers developed attention-free architectures competitive with transformers
“So one of my colleagues, Chris Ray and his students have developed other architectures, which are actually at smaller scales, competitive with transformers. And actually don't require the central operation of attention.”
Apr 25, 2023 positive
Srinivas: Academic AI researchers should pursue radical alternatives to transformers
“I think it's best to look for alternatives to the transformer, alternatives to like language models, that sort of radical directions, then Trying to improve them because there's so much incentive for the existing companies to do that.”
Apr 25, 2023 positive
Self-Attention Brought GPU Parallelism to Sequence Modeling
“The insight here was, Hey, you can use the same attention thing to like, look back at the past of the sequence that you're trying to produce. And you know, the beauty is that the, that It runs great on on GPUs and CPUs, and it's kind of parallel to, like, how …”
Apr 25, 2023 positive
Transformers Beat Recurrent Models by Processing Whole Sequences at Once
“The magic of transformer kind of like convolutions is that you get to process the whole sequence at once. I mean, it still talks, you know, it's still a function of like the, you know, the predictions for the later words are dependent on what the earlier words…”
May 1, 2023
Valenzuela: It takes 12-24 months to understand new AI breakthroughs
“The moment something gets released, like, let's say transformers or a particular piece of technology that you think would be interesting or could be worth experimenting with, I think it takes a collective set of months, like, 12, 24 months sometimes to underst…”
Aug 24, 2023 neutral
Uszkoreit: Transformer breakthrough was driven by accelerator hardware fit
“And if you want to look at, say, the biggest differences, for example, between the transformer, as it was described in the attention is all you need paper, and some of its ancestors, like this decomposable attention model, the big difference is just that the t…”
Sep 14, 2023 positive
Polosukhin: Market full of Transformer-optimized accelerators will launch by 2024
“And so like, we're going to have a, you know, a market full of hardware accelerators, which are still optimized for transformers, or at least like similar structured architectures hitting the market like this year and next year.”
Sep 14, 2023 bullish
Oct 26, 2023 bullish
Gil: AI capabilities represent a major discontinuity, not a linear progression
“I feel like they're in general, people are viewing AI as this continuum where it's like, it's a CNN, RNN, and now we have transformers and it's just a straight line. And instead, obviously it's a big discontinuity in terms of capabilities. And I think most peo…”
Nov 2, 2023 positive
Dec 14, 2023 bearish
Dec 21, 2023 bearish
Apr 18, 2024 negative
May 16, 2024 positive
Suno builds on transformers and focuses innovation on audio tokenization
“We don't make it a secret that these are just transformers. This is somewhat our backgrounds doing text before, but also transformers scale nicely. A lot of work ends up being done for you by the open source text community, which is always really nice. We can …”
Jun 27, 2024 bearish
Gu: Transformers struggle significantly on raw pixel or audio waveform data
“People think that like you can throw a transformer at like anything and it just works. Actually it doesn't really like if you try to throw it at like the raw pixel level or the raw sample level and in audio waveforms I think it doesn't work nearly as well.”
Jun 27, 2024 positive
Gu: State Space Models can be applied to almost all data types
“So it really can be applied to pretty much everything. So just like kind of Transformers, these are applied to everything. So can these sort of models over the course of research over a few years, we kind of realized that there are different advantages for dif…”
Jun 27, 2024 neutral
Aug 30, 2024 positive
Sep 5, 2024 neutral
Karpathy: RoPE is the only major transformer architecture change in five years
“The transformer hasn't changed that much. You know, we've added the rope positional and the rope relative positional encodings. That's like the major change. Everything else doesn't really matter too much. It's like plus three percent on a small few things. Bu…”
Sep 5, 2024 positive
Karpathy: Transformers are a more efficient system than the human brain
“I think transformers are actually better than the human brain in a bunch of ways. I think they're actually a lot more efficient system. And the reason they don't work as good as the human brain is mostly data issue, roughly speaking, is the first order approxi…”
Sep 5, 2024 positive
Karpathy: Clean AI scaling laws are a property of transformers, not LSTMs
“When people talk about the scaling loss in neural networks, the scaling laws are actually a to a large extent of a property of the transformer. Before the transformer, people were playing with LSTMs and stacking them, etc. You don't actually get like clean sca…”
Sep 5, 2024 bullish
Karpathy: No single technical bottleneck is blocking robotics, just data grunt work
“I don't know that there's, like, any individual impediments that I'm, like, really familiar with. I just think it's a lot of grunt work. A lot of, like, the tools are available. Transformers are this beautiful, like, blob of tissue. You can just get just arbit…”
Oct 24, 2024 negative
Dolgov: Off-the-shelf AI models cannot achieve human-beating driverless safety records
“The power of transformers, the power of realism is mind-blowing, right? So with just a little bit of effort, you get something on the road, and it works. You can, you know, drive, I don't know, 1000 of miles, and we just, it will blow your mind. But then is th…”
Mar 20, 2025 positive
May 15, 2025
May 15, 2025 positive
Jain: Enterprise search requires transformer models due to scarce user signals
“On the web, even if you don't have good semantic understanding, there is so much that you're going to learn from people's behavior because, you know, you have a billion people, you know, coming and using your product. In the enterprise, you don't have that lux…”
Oct 31, 2025 bullish
Sep 3, 2026
Haas: AI chip startups need supply chain access, not just great design
“So long as the transformer is the unit of energy relative to how you generate AI training and AI inference by design, it is a, it is, it's very compute intensive, it's very memory intensive. So if you think about that, that's going to drive a lot of demand on …”