transformer architecture

“The other main contributor, I think, to the success of this architecture was optimism and hope. So suddenly you were in a situation where, for whatever reason, a bunch of things that people tried with this started to work, and then more started to work, and th…”

“And so this is, I feel like, a stronger version of that in some sense, where you have the underlying compute architecture and the most important model reinforcing each other in a way that kind of locks both of them in.”

“So it's better to think about it in terms of compute efficiency rather than in terms of, can it get there at all? I think at this point, the answer is obviously yes.”