Transformer
also referred to as: transformers
41 statements across 25 episodes · 14 bullish · 10 bearish · 24 people on the record · first statement Jun 20, 2023 by George Hotz · said 5 times in 3 episodes since 2025 · across every show →
Mentions by year
brought up most by Yi Tay (2), Elie Bakouch (1), Chris Manning (1)
tap a year for its mentions
2026 4 mentions in 2 episodes 2 per episode
Everything said about Transformer, oldest first
Jun 20, 2023 negative
Hotz: Qualcomm SNPE cannot run transformers due to missing outer product ops
“Qualcomm's S and PE can't run transformers for this reason. So most matrix multiplies in neural networks are weights times values, right? Whereas you know, when you get to the outer product in in transformers, well, it's waste times weight. It's a, it's values…”
Aug 3, 2023 positive
Aug 3, 2023 bullish
RNNs will outperform Transformers in batch generation and long sequences
“I am personally bullish on, on, on RNNs. I think RNNs they don't, they essentially summarize the past into a state vector. They have fixed size, so the size doesn't grow with the history. So that means that you don't need as much memory to keep around all the …”
Aug 3, 2023
Hardware and software co-evolve to favor dominant AI architectures
“There is this feedback loop where somehow The model architectures that take advantage of hardware become popular, and the hardware will also kind of evolve to optimize a little bit for that kind of architecture, and software framework software frameworks will …”
Aug 31, 2023 neutral
Bo Peng Created RWKV on EleutherAI Forum to Parallelize RNNs
“Blink, or Bopeng is the actual name decided basically as an individual, literally at the illiterate AI forum, decided that, hey I think we can modify recurrent neural networks, no, neural networks, based on the Apple paper, the light attention that I showed pr…”
Aug 31, 2023 positive
Fine-Tuning Transformers Requires Only 1,000 to 10,000 Data Examples
“Post-Transformer it's literally, you probably need only, like, a thousand or 10,000, enough, like, data that you can literally get an intern a few weeks to just get it done, and you have a working model. It may not be that great, but frankly, every piece of da…”
Aug 31, 2023 bullish
Aug 31, 2023 bullish
Aug 31, 2023 bearish
Aug 31, 2023 negative
Aug 31, 2023 negative
Cheah: Pre-Transformer Academic Neural Network Research Is No Longer Relevant
“Frankly, almost everything that is, that matters, Ah, was basically in the past four years. Like, there were a lot of things that fit in academics that were before that, and you know, and they were mostly dealing with models that were under a billion parameter…”
Feb 8, 2024 bearish
Mar 14, 2024
Apr 24, 2024 neutral
Apr 27, 2024 neutral
May 31, 2024 positive
Huang: Google's internal AI tooling was far superior to competitors
“Google was using AI for systems before everybody else too, right? They invented a transformer, and their internal set of tooling was just so far superior to everything else. Like, it's really hard for people to go back after seeing that.”
Jul 23, 2024 neutral
Scialom: Fixed compute per token in transformers forces models to generate extra tokens to think
“We are lacking of flexibility for pre-training architecture, transformers, where We spend the same amount of compute per token. And so, because of that, how can you, like, mitigate this? By generating more tokens, so more thoughts, more compute, because you ha…”
Sep 21, 2024 positive
Nov 11, 2024 negative
Polu: Transformers failed at code fuzzing because they are too slow
“We're trying to apply transformers to code fuzzing. So code fuzzing, you have kind of an, sorry, an algorithm that goes really fast and tries to mutate the inputs of a library to find bugs. And we try to apply a transformer to that and do reinforcement learnin…”
Nov 11, 2024 positive
Polu: Combining LLMs with formal math pairs creativity with proof verification
“Transformers are very creative, but yet they do mistakes. And formal math systems are the ability to verify a proof. And the tactics they can use to solve problems are very mechanical. So you miss the creativity. And so the idea was to try to explore both toge…”
Dec 24, 2024 positive
Cheah: Hybrid SSM-transformer models outperform pure baselines of both
“None of us understand why a hybrid with a state-based model, the RWA state space, and transformer performs better than the baseline of both. It's like when you train one, you expect, and then you replace, you expect the same results. That's our pitch. That's o…”
Jul 2, 2025 neutral
Morris: ChatGPT could likely have been built using RNNs instead of Transformers
“And I think like, we honestly probably could have gotten this with RNNs. I know like the scaling laws paper shows that RNNs have worse curves for scaling, but probably people would have been like, I bet you could have built chat GPT with a very sophisticated R…”
Jul 2, 2025 neutral
Jul 28, 2025 negative
Mohan: Off-the-shelf serving frameworks leave significant FLOP utilization on the table
“The open source serving. Offerings are just, I will say not great in that they aren't customized to transformers and these kinds of workloads where I have high latency and I want to like batch requests and I want to batch requests while keeping latency low. Bu…”
Aug 6, 2025 neutral
Swyx: Most AI researchers are scaling Transformers, not alternative architectures
“I think most people that I talk to are not seriously pursuing alternative architectures. There are some notable exceptions, primarily together with the Mombard architecture recursal with RWKV. There's like the XLSTM that was created by a separate Hogwriter. An…”
Aug 18, 2025
Sohmers: Transformer inference is memory-bound with a 1:1 FLOP-to-byte ratio
“And on the other side of this chart, you have the case of a transformer where when you're actually, you know, doing attention, Or really just any case where you're doing, you're fundamentally doing matrix vector multiplication rather than matrix matrix multipl…”
Aug 29, 2025 neutral
Morcos: The Transformer is just one of many equivalently good architectures
“And one of my like more controversial viewpoints, I think, is that I think the transformer is a great advance to be sure, but I think it's one of a very large Set of equivalently good architectures that we could have found. And there are many, many ways we cou…”
Sep 23, 2025 negative
Bachman: Models claiming 256k+ context use windowed transformers, discarding data
“Anybody who says they're using a transformer
With a context length of, you know, 256,000 or more, they're not using a true transformer.
What they're using is a windowed transformer that essentially throws out a huge amount of its information at various layers …”
Sep 23, 2025 neutral
Bachman: Compute-optimal models on internet text don't need long context
“In general, most internet text has mostly short-term structure. There's just not that much value in capturing long-term structure, and so compute optimal models on internet text actually don't have that long context, and so, of course, you're perfectly fine us…”
Nov 25, 2025 neutral
Jan 23, 2026 neutral
Yi Tay: Today's Models Likely Couldn't Invent the Transformer from Pre-2015 Data
“Even today's models, they might not even be able to invent the transformer. Like, if you freeze the time at a certain time, and even you bring the time, I mean, the model is a transformer, so I just say there's no, assuming there's no leakage.”
Jan 23, 2026 bullish
Yi Tay: The Architecture That Achieves AGI Will Still Be a Transformer
“It will be a transformer, I think. Like people, it depends on what you call it, but I think unless the paradigm shifts completely, which is, I mean, as a scientist, you cannot like completely say no to like that, this would never happen. But my feeling is that…”
Jan 23, 2026 neutral
Yi Tay: The AI Industry Is Stuck in a Transformer Local Minimum
“So now we are like in this local minima of like transformers, everything, everything, right? Maybe it's not easy to like get totally out Of this, because also a lot of people's investment optimization have been done. So the things that play well needs to play …”
Jan 23, 2026
Yi Tay: Gradient descent learning paradigm is AI's bottleneck, not architecture
“It's not architecture itself. That's, that there's a problem that we, that is more of like the learning paradigm itself rather than the architecture itself. I think the architecture is just basically like the interface between the learning algorithm and the to…”
Feb 12, 2026 positive
Apr 2, 2026 positive
Manning: Transformer internal weights can act as joint representations for world models
“I'm not actually convinced that's right, because although the token production is this autoregressive process that's heading, you know, left to right, I guess don't have to be left or right, but anyway, in sequence of tokens, we could have right to left Arabic…”
Apr 3, 2026 positive
Jun 1, 2026 neutral
Ethan He: Training transformers directly on raw image pixels is impossible
“If you're trying, if you can technically, theoretically train image or video models on pure pixels, but the problem is that the, it's a lot of tokens. So like one image, like it's a thousand by a thousand is like one million tokens, one million pixels. It's im…”
Jun 1, 2026 negative
Aug 26, 2026 positive
Fourier neural operators scale quasi-linearly, avoiding transformers' quadratic complexity
“If we were to use transformers and we require a very high resolution, it would become untenable because of the quadratic complexity and all to all connections. On the other hand, if you did that with Fourier transforms, we have like quasi linear complexity and…”
Aug 26, 2026 bearish
Transformers will never scale to high-resolution 4D physics simulations
“So forget ever having a transformer for anything of this scale. All of the world's compute will not be enough. And first of all, they all have to be co-located to be able to ever do this. So that's why we need other architectures.”