Transformer

“Conductor is, like, twice as expensive. So, like, basic aluminum, steel-reinforced conductor. Transformers are two-plus times more expensive. Switchgear is two times more expensive. Gas power plants are two to three times more expensive.”

“So long as the transformer is the unit of energy relative to how you generate AI training and AI inference by design, it is a, it is, it's very compute intensive, it's very memory intensive. So if you think about that, that's going to drive a lot of demand on …”

“There isn't going to be less code. The code just becomes less valuable throw away.”

“If we were to use transformers and we require a very high resolution, it would become untenable because of the quadratic complexity and all to all connections. On the other hand, if you did that with Fourier transforms, we have like quasi linear complexity and…”

“So forget ever having a transformer for anything of this scale. All of the world's compute will not be enough. And first of all, they all have to be co-located to be able to ever do this. So that's why we need other architectures.”

“And I would say the original sin of Transformers is that you've taken this extremely fundamentally memory bound layer and juxtaposed it right next to a compute bound layer. It is very difficult to have a single chip that is good at both compute operations and …”

“What transformers really did well is that they scaled. Transformers make no such human prior. Transformers just say, well, there's gonna be a pattern in the sequence of data, and if there is a pattern, I'm gonna find it.”

“The data is really the hardest part of model training. Because you need to source it. It's so dirty. Nobody wants to do that shit. Nobody wants to hire all these people to generate data and then, you know, turn that into basically data plus GPUs equals model. …”

“At least at the time Transformers came out, BERT came out and all that, nobody really thought this was a this was a path to emergent reasoning capabilities, a path to AGI itself.”

“Those industries have historically have not added much capacity for the last decade or so ago, and they've suddenly experienced a demand shock. And it takes years before you can add capacity to produce more turbines and transformers.”

“Using both of these together was actually better than using either one on their own. And that is independent of the speed benefit. That is just the model is smarter.”

“Anything you want to order today is probably going to take three plus years in the case of gas turbines, probably more like five years to be able to get your hands on. Anything you order today, whether it's a turbine or a transformer or just the, you know, alu…”

“We've also seen new ideas, but mainly since Transformers, it's been much better hardware, many more resources better engineering, and many more talented people.”

“If you're trying, if you can technically, theoretically train image or video models on pure pixels, but the problem is that the, it's a lot of tokens. So like one image, like it's a thousand by a thousand is like one million tokens, one million pixels. It's im…”

“So people actually have tried that, but the main challenge is the latent space for the MP four tokens are not, we're not very comprehensible for the models. It's extremely hard to train on that.”

“And if you look at the coauthors of that paper has a ton of coauthors, not a single one commercialized the transformer at Google. They all, every single one had to leave and do it elsewhere.”

“In a one-shot basis. It's like literally that we know a theoretical lower bound that for comparison sort, you can't do better than n log n steps. And if I have a list that's 31 characters or elements long, and my transformer is 30, I run out of steps to do com…”

“Recursion advantage now gives you a bunch of advantages over transformers where rather than having, you know, 500 or a thousand or a million or whatever transformer layers and having tons and tons of parameters, you get compute depth basically without this par…”

“BERT and MUM, people underestimate how much, because we measure search quality so religiously, some of the biggest jumps in search quality in that period where search went ahead of everyone else was because of BERT and MUM. We built transformers and used it im…”

“One was transformers. That were originally successful for large language models, like you heard about Anthropic and OpenAI those started to have an impact on self-driving technology and robotics, and we thought that was really interesting, and then on top of t…”

“I think the real story is it was the Alex net basically breakthrough in like 2013. That was the real knee in the curve. And then it was obviously the transformer breakthrough in 17.”

“I'm not actually convinced that's right, because although the token production is this autoregressive process that's heading, you know, left to right, I guess don't have to be left or right, but anyway, in sequence of tokens, we could have right to left Arabic…”

“And I was like, I don't know if I want to go with this team. It's just like, they're doing something random. Like who, like everybody's doing LST. I'm like, why should I go and work with like a group of people who are working on this like random architecture, …”

“Even if this is not, like, the only architecture that would be, like, in a multi-model, but it made it really simple to train these models, like, natively, because you have, like, a single architecture and you can have all the modalities in, during training.”

“We trained these models and we found that the transformer got the precise Bayesian posterior down to 10 to the power minus three bits accuracy. It was matching the distribution perfectly. So it is actually doing Bayesian in the mathematical sense, given a task…”

“Transformer does everything. Mamba does most of it. LSTMs do only partially, and MLPs fail completely.”

“The five to seven years is the timeline to get new gas turbines, if you're ordering them, and it's close to the timeline to get new transformers and other, and switchgear and stuff like that. Like, we're all in the, like, three to five or maybe seven year time…”

“The off-grid option in some ways maybe shortcuts the actual supply chain bottlenecks on the generation equipment side, at least to some extent relative to the other options. But I agree with you, there's still a bunch of other pieces of equipment, transformers…”

“Our strong belief is like state space models are not like that big a leap over transformers.”

“The best way to make power electronics Systems that involve isolation more affordable is go up in frequency, because to get isolation, you basically need to use a transformer of some type, and transformers become smaller as you go up in frequency. It's just a,…”

“Transformers are not really designed to run at their rated power, you know, as long as they do in these Desert power plants, you know, where they're, first of all, very hot, because they're sitting in the sun, and second of all, because they're running at name…”

“Transformers similarly gave you a 10 X to a hundred X improvement in, you know compute cost to a given quality level versus say LSTMs at the time.”

“Transformers are not the end state of AI, right? AI is Embarrassingly stupid compared to many of the examples we are carrying around in our heads, both in terms of what it can achieve, but also how much power it uses to do it.”

“The transformer moment for reasoning and choosing action and being able to plan at different levels of granularity. We're still far away from doing that.”

“Even today's models, they might not even be able to invent the transformer. Like, if you freeze the time at a certain time, and even you bring the time, I mean, the model is a transformer, so I just say there's no, assuming there's no leakage.”

“It will be a transformer, I think. Like people, it depends on what you call it, but I think unless the paradigm shifts completely, which is, I mean, as a scientist, you cannot like completely say no to like that, this would never happen. But my feeling is that…”

“It's not architecture itself. That's, that there's a problem that we, that is more of like the learning paradigm itself rather than the architecture itself. I think the architecture is just basically like the interface between the learning algorithm and the to…”

“So now we are like in this local minima of like transformers, everything, everything, right? Maybe it's not easy to like get totally out Of this, because also a lot of people's investment optimization have been done. So the things that play well needs to play …”

“If anything, they're still ticking slightly upward. And as much as there's energy and attention and noise being paid to, we need to build stuff here, we're not.”

“Gas turbine manufacturing is even more capital intensive. It's currently more of an oligopoly than transformers are.”

“At least for some tasks, I'm pretty confident that the transformers will be highly suboptimal.”

“We actually use transformers not to model the spatial, the interactions in weather over time, like the sequence of text, but in space.”

“And because all I guess for the most part, all currently deployed chatbots are based on transformers or transformer adjacent technologies. They're all vulnerable to Prompt injection, jailbreaking, forms of adversarial attacks.”

“I believe so. There's groups doing research on the model architecture side, for sure, within Google and within DeepMind”

“Transformers are really they're a big innovation because they made the constructs of the GPU work extremely well.”

“One point was, of course, the Transformers when it started, but the other point was reasoning models.”

“Transformers are actually not a model of sequences. A transformer is natively a model of sets.”

“Reasoning is the biggest paradigm shift in AI architecture since the transformer.”

“In my reporting, like I've come across many, many projects like housing projects that are completely on hold for 18 months because they're missing this one object that they need before they can Power the area.”

“I think there's going to be a wave of upstart companies, whether they're doing traditional oil-filled transformers or solid-state stuff like Heron is doing. I think they're going to fill in a lot of the supply gap in the next few years”

“The entire AI revolution that we are in right now is predicated by the invention of the transformer out of the Google brain team in 2017.”

“It is fair to say that 2017 begins the five-year period of Google not sufficiently seizing the opportunity that they had created.”

“So in June of 2018, OpenAI releases a paper describing how they have taken the transformer and developed a new approach of pre-training them on very large amounts of general text on the internet, and then fine tuning that general pre-training to specific use c…”

“I don't think that's true. I think we haven't yet really found anything that transformers haven't been able to model provided sufficient data and sufficient compute.”

“Anybody who says they're using a transformer With a context length of, you know, 256,000 or more, they're not using a true transformer. What they're using is a windowed transformer that essentially throws out a huge amount of its information at various layers …”

“In general, most internet text has mostly short-term structure. There's just not that much value in capturing long-term structure, and so compute optimal models on internet text actually don't have that long context, and so, of course, you're perfectly fine us…”

“It wasn't then commercialized very quickly within Google. It wasn't scaled up very quickly within Google. A lot of that work had to be done elsewhere and years later.”

“And one of my like more controversial viewpoints, I think, is that I think the transformer is a great advance to be sure, but I think it's one of a very large Set of equivalently good architectures that we could have found. And there are many, many ways we cou…”

“Google invented the Transformer and published the paper in June of 2017.”

“And on the other side of this chart, you have the case of a transformer where when you're actually, you know, doing attention, Or really just any case where you're doing, you're fundamentally doing matrix vector multiplication rather than matrix matrix multipl…”