transformer, every mention
33 scenes · ← back to transformer
tap a year for its mentions
every year anyone David Rosenthal 23Ben Gilbert 10
Verbatim, from the transcripts: the passages where transformer comes up
Google Part III: The AI Company. Google is amazingly well-positioned... will they win in AI? (Audio) · Acquired
- ▶ 1:52:14 David Rosenthal Because this is the Transformer paper.
- ▶ 1:52:17 Ben Gilbert Okay, so where did the Transformer come from?
- ▶ 1:53:26 Ben Gilbert And this was the year before the Transformer paper would come out.
- ▶ 1:53:29 David Rosenthal Before the Transformer paper, yes.
- ▶ 1:57:09 David Rosenthal They decide that they're gonna call this new technique the transformer.
- ▶ 1:57:27 David Rosenthal That's not not why they named it the transformer.
- ▶ 1:57:30 David Rosenthal And it's taking in the giant corpus of text and storing it in a compressed format, right?
- ▶ 1:58:05 David Rosenthal And it's a good thing he did, because before Gnome joined the project, they had a working implementation of the transformer, but it wasn't actually producing any better results than LSTMs.
- ▶ 1:58:16 David Rosenthal Gnome joins the team, basically pulls a Jeff Dean, rewrites the entire code base from scratch, and when he's done, the transformer now crushes the LSTM-based Google Translate solution.
- ▶ 2:00:27 David Rosenthal Yeah, there's an elegance to the Transformer.
- ▶ 2:01:45 David Rosenthal It is a false narrative out there that Google did nothing with the transformer after the paper was published.
- ▶ 2:03:08 David Rosenthal And also, of course, within a couple years, all eight authors of the Transformer paper had left Google to either start or join
- ▶ 2:03:48 David Rosenthal With the Transformer, yes.
- ▶ 2:03:53 Ben Gilbert And does anyone think the Transformer's a big deal over there?
- ▶ 2:04:02 David Rosenthal Right after Google publishes the Transformer paper, in September of 2017, Elon gets really, really fed up with what's going on at OpenAI.
- ▶ 2:05:56 David Rosenthal The sort of panic that this throws OpenAI into is the catalyst that makes them reach for the Transformer and say, all right, we gotta figure things out.
- ▶ 2:06:11 Ben Gilbert I don't know if during this personal tension between Elon and Sam, if they had already decided to go all in on Transformers or not.
- ▶ 2:06:25 Ben Gilbert Transformers, language models, we're going all in on that.
- ▶ 2:07:33 David Rosenthal So in June of 2018, OpenAI releases a paper describing how they have taken the transformer and developed a new approach
- ▶ 2:08:14 Ben Gilbert Which we should say is right around the same time as BERT and right around the same time as another large language model based on the Transformer out of here in Seattle, the Allen Institute.
- ▶ 2:20:33 David Rosenthal So, Noam Shazir, incredible engineer, re-architected the Transformer, made it work, one of the lead authors of the paper, storied career within Google, has all of this sway, should have all of this sway within the company.
- ▶ 2:20:48 David Rosenthal After the Transformer paper comes out, he and the rest of the team are like, guys, we can use this for a lot more than Google Translate.
- ▶ 2:20:59 Ben Gilbert Are you about to read the Transformer paper?
- ▶ 2:21:07 David Rosenthal We plan to extend the transformer to problems involving input and output modalities other than text
- ▶ 2:21:33 David Rosenthal Hey, I think this is going to be so big, the transformer, that we should actually consider just throwing out the search index
- ▶ 2:21:42 David Rosenthal And the 10 blue links model and go all in on transforming all of Google into one giant transformer model.
- ▶ 2:21:51 David Rosenthal And then Gnome actually goes ahead and builds a chatbot interface to a large transformer model.
- ▶ 2:25:17 David Rosenthal Gnome knew, and people in Google knew, that you could make a chatbot interface to a transformer-based LLM, and that was a really compelling product.
- ▶ 2:57:56 Ben Gilbert 2017, obviously, the Transformer comes out.
- ▶ 2:57:58 Ben Gilbert They incorporate some learnings from the Transformer, especially around prediction and planning.
Google Part II: Alphabet (Audio) · Acquired
- ▶ 3:40:43 David Rosenthal Andrew Ng from Stanford, Kwok Lee, Oriol Vinales, and oh yeah, in addition to all of those people, the authors of the Transformer paper because Google invented the Transformer and published the paper in June of 2017.
- ▶ 3:41:10 David Rosenthal When I was talking to folks in the research for this episode and AI came up, one of them said, you know, I have to remind people when I'm talking to partners out in the ecosystem that the T in chat GPT stands for transformer and that we…
Nvidia Part III: The Dawn of the AI Era (2022-2023) (Audio) · Acquired
- ▶ 45:14 Ben Gilbert Google Brain's Transformer had blown open a new frontier where AI could improve endlessly.