The Exchanges

Q Can you quickly describe sort of the difference between an RNN and, uh, in a transformer based or attention based model?

A Yeah, sure. Um, so, um, Okay. So like the recurrent neural network is like the sequential computation where every word, you know, you read the next word and you, you kind of compute your current, you know, state of, uh, of, of your brain based on the old state of your brain and the, um, you know, and what this, this next word is, and then you, you predict the next word. So you have this very long sequence of computations that has to be, Um, you know, executed in order so that, you know, the magic of transformer kind of like convolutions is that you get to process the whole sequence, uh, at once. I mean, it, it still talks, you know, it's still a function of like the, you know, the, the predictions for the later words are, uh, dependent on what the earlier words are, but it happens in like a constant number of steps where you, where you get to To, to take advantage of like this parallelism. If you can look at like the whole thing at once and like, that's what modern hardware is good at is parallelism. And now you can use the length of the sequence as your parallelism and everything works, you know, super well. Um, attention itself, it's kind of like you're You're creating this big, um, like key value associative memory where you're, you're like building this big table, like with one entry for every, for every word in the sequence. And then you're kind of looking, looking things …

AI assessment note: “the magic of transformer kind of like convolutions is that you get to process the whole sequence”

Q So what do people, what do people want? Like, do they do like their friends? Do they do fiction? Do they do entirely new things?

A Yeah. I mean, there's like a lot of, um, you know, there, there's a lot of, you know, role-playing, like role-playing games are big, you know, like, you know, like text adventure where it's just like making it up, uh, uh, as it goes, there's a lot of like video game characters and anime and there's, um, you know, uh, you know, some amount of people talking to public figures and influencers and like, You know, like, I think a lot of people have these existing parasocial relationships where there's, you know, they've got characters they're following like on, uh, on TV or some, uh, you know, or internet or influencers or, or whatever. And, um, and so far they just have not had the experience of, okay, the, now this character responds. Cause like, you know, it's, it's always something you can watch, or maybe you're in like a, you know, Thousand on one, like, fan chat or something where, like, this VTuber will, like, write back to you, like, once, you know, once in an hour or something, but, like, now they get the experience of, oh, like, I can just create a version of this privately and just, like, uh, you know, just, like, uh, talk to it, and, uh, it's pretty fun. We also see, like, a lot of You know, people using it cause they're, you know, they're lonely or, uh, or, uh, or troubled and need someone to talk to. Like so many people just don't have someone to talk to. So like, you …

AI assessment note: “there's a lot of like video game characters and anime and there's”

Q You have some users that are on the service, like many hours a day. Um, like how do you think about your target user over time and like, you know, what the usage patterns you expect to be are?

A Um, we're gonna just leave that up to the user. Like, uh, our, our aim has always been like, get something out there and let users decide, you know, what, what they think it's, uh, what they think it's good for. And, you know, we see like, yeah, yeah. Like, uh, somebody who's on the site today is active for about two hours on average today. Uh, that's of people who send a message today, which is, which is pretty wild. That's pretty wild, but like, um, Yeah, it's, it's definitely, you know, it's a great metric, like that people are finding some sort of value in it. And as I said, it's really hard to pin down exactly what that value is, you know, but because it's, it's really like, uh, you know, a big mix of things, but like I, our goal is like, make, make this thing more useful to people and let people kind of customize it and decide, uh, what they want to use it for. If it's, uh, if it's It's, you know, brainstorming, or help, or information, or, uh, or fun, or like emotional support. Like, um, you know, let's, let's, uh, just get it, get it into user's hands and, and see what happens.

AI assessment note: “we're gonna just leave that up to the user. Like, our aim has always been”

Q Do you want to talk a little bit about Lambda and your role with it and, you know, how that led eventually to character?

A Yeah, uh, sure. Yeah, I guess, um, yeah, my co-founder Daniel, uh, Daniel de Freitas, he's like the, you know, scrappiest, uh, most, uh, you know, hardworking, really, you know, smartest guy, you know, he's kind of been on this lifelong mission to build, um, chatbots. Like you, like, since like, he was like a kid in Brazil, he's like always, you know, been trying to build chatbots. So like he, uh, came to join us at Google Brain because I think he had Read some, some papers and, uh, figured that this neural, um, neural language model technology would be like, you know, something that could actually generalize and, you know, uh, and build something truly open domain. So, and, and like he, um, yeah, didn't, did not get a lot of head count. Like he started the thing as like a 20% project where like people are encouraged to, uh, spend 20% of their time, like doing whatever they want. So, um, and then he just, like, recruited, like, an army of, like, 20% helpers who were, like, ignoring their, uh, their day jobs and, like, actually just, uh, you know, helping them with the system, and he, like, went as far as, like, um, going around and, like, uh, panhandling people's TPU quota, like, and he called this project Mina, because he, I guess it came to him in the dream, and, like, at some point of, like, looking at the scoreboard and was like, What is this thing called Mina and why does …

AI assessment note: “my co-founder Daniel, uh, Daniel de Freitas... kind of been on this lifelong mission”

Q ones are, um, these multimodal language models. So it's things like ChatGPT or what you're doing a character. Um, I've also been surprised by some of the applications into things like AlphaFold, the protein folding efforts that Google did where it actually worked in an enormously performant way. Are there any application areas that you found really unexpected relative to how transformers work and relative to what they can do?

A Oh, um, I've just had my head down in, uh, in language like that. Like here you have like something that like a problem that like can do like anything. Like I want this thing to be good enough. So I just ask it, like, how do you cure cancer and that like invents a solution? Um, and you know, like, so, so I've been totally ignoring like what everybody's been doing in, uh, in all these other modalities where like, I think a lot of the early successes in, in deep learning Have been like in images and people are like all excited about images and I kind of like completely ignored it cause like, you know, an image is worth a thousand words, but it's like a million pixels. So like the text is like a thousand times as dense. So like kind of big, uh, big, uh, text, uh, text nerd here, but, um, you know, it's very exciting to see it, uh, it take off in, you know, in all these other modalities as well. And, you know, those things are going to be great. It's like, Super useful for, uh, building products that people want to use. Uh, but I think that a lot of the core intelligence is going to come from, from these text models.

AI assessment note: “I've been totally ignoring like what everybody's been doing in, uh, in all these other modalities”

Q Why do you think we do this podcast?

A Right. Like there's like order, like ten billion people, like, uh, you know, producing like a thousand, you know, like, I don't know, 10,000 words a day. I mean, that's like, that's like a lot of words that, you know, and pretty soon a lot, many of those people will be doing a lot of that, that talking, uh, to AI systems. So I, I have a feeling like a lot of data is going to, um, find its way into some AI systems. Um, But, uh, I mean, in privacy preserving ways, I would, uh, I would hope. And then the, um, you know, the, the data requirements tend to go up like with the square root of the amount of computation, because you're going to train a bigger model and then you're going to, um, throw more data at it. So, uh, you know, I, I, I think I'm not that worried about Coming up with data and I feel like we could probably like just generate some more with the AI.

AI assessment note: “a lot of data is going to, um, find its way into some AI systems”