The Exchanges

Q Now I would love to start with some context because few people spend 20 years at Google in the height of Google's scaling and trajectory. First, I want to go back to the beginning. I heard there's a story to your joining. What happened spelling corrector? Can you give me the story?

A Um, yeah, that was, uh, yeah, that was like the first project that I, uh, that I worked on at Google. Yeah. I guess at the time we You know, Google had a spelling corrector that was, uh, you know, it was some third party software. It was, uh, you know, based on maybe what you'd find in a word processor at the time. So there was like some human compiled dictionary of maybe about 50,000 words and any word that wasn't in the dictionary that was in the query, it would say, you know, that you mean such and such. And this worked great for spelling correction. It was like absolutely terrible for web search because People search for such a wide diversity of things on web search. Like most of them are just not in the dictionary. So like you'd search for turbot, like turbotax, and it would say, did you mean turbotax? Like, and people just learn to ignore the thing. So first project, like we were just looking at like, why are people like not happy using Google and like spelling correction was like the, you know, number one, uh, number one issue. So I was like, okay, let me, let me help out with this. And, you know, there was someone working on this, Paul Buhite. Who, uh, you know, who's, uh, you know, gone on to, uh, do a lot of, uh, a lot of illustrious things in his, uh, career. He's also one of our investors here at, uh, uh, Character, but, uh, he was going on, uh, on vacation for a co…

AI assessment note: “that was like the first project that I worked on at Google”

Q What did you believe that you turned out to be wrong on?

A You know, when I started getting into deep learning, I, uh, you know, Um, around 2012 had a bunch of early, uh, failures trying to, um, do sparse computation. You know, I was like, okay, like you must be able to do something, you know, better and more efficient by building a sparse network. And that was, that was so wrong because I, uh, I did not understand that the reason this whole field is working so well is because Now we have this magic hardware that's great at these dense matrix multiplications. And so you can do them like orders of magnitude faster than you can do anything that involves poking around in memory. And there was no one there to like explain that to me when I got started with deep learning. Um, so, okay, as soon as I sort of understood that part of it, it's like, okay, let's do sparsity, but let's build it out of these dense building blocks. So it'll, so that it'll run fast and then, you know, publish this, uh, mixture of, uh, Experts, a sparsely gated mixture of experts idea that's, uh, only now getting, like, a lot of adoption, but, you know, that, that was back in, uh, uh, 2016, and then have had, like, a string of hits ever since, which I will attribute to, uh, to divine intervention, but also to, um, you know, to, to understanding, like, the, the hardware mechanics and the, uh, sort of quantitative computation aspects of the field.

AI assessment note: “building a sparse network. And that was, that was so wrong”

Q When you get up in front of the company, and you share, you, there's many people in the team now, what do you chest pump as the mission?

A I like this, Sort of motto of, you know, a billion users inventing a billion use cases, because that, that's sort of the superpower of this technology. And, you know, it sort of puts our company in, you know, in the right place of, we can't really, you know, guess, you know, what are the best uses, uses of this technology. And, you know, we've just You know, observe time and time again, like you put one thing out there and that's not really what people want. And somebody else out there, like find something better to do with it. We put up as an example, like a psychologist character, like maybe, maybe you want to, you know, talk to something and feel better, you know, and you know, that gets a little bit of use, but then what we hear a lot more from users is like, I'm talking to a video game character who's now my new therapist, and this makes me feel better. We had no idea that was, that was going to go on. Um, but like, and then, you know, there's this huge, like, use case in, like, some mix of, like, entertainment and companionship and emotional support. We were totally not experts in this stuff. Like, you know, like, our job is just to put out something general And, like, just respect the agency of, like, our users, of everybody out there to, uh, you know, to, to do what they want with this stuff.

AI assessment note: “a billion users inventing a billion use cases”

Q You said the more and more you do it, kind of, the better it gets, and the better responsiveness and accuracy it gets. I, I'm always concerned, not concerned, but just, like, Um, questioning, because you hear so much. What's more important? Is it the size of the data, or is it the size of the model?

A Yeah, probably the size of the model is the, is the bigger challenge. We can get a lot of, we can get a lot of data, but like, really, actually the number one thing that's important is how much computation you do to train it. So you want to train a bigger model, and you want to train it for longer. The two things are both important, but What the real constraining factor has been is How much, how many operations of computation it takes to train it, because if you make it bigger and you train it for longer, both of those multiply into how long the thing takes to train. So people have been building better and better, uh, essentially supercomputers to, you know, to train these models.

AI assessment note: “probably the size of the model is the, is the bigger challenge.”

Q What are one to two of the biggest takeaways from your time? 20 years is a long time. How did it impact you?

A I'd say one of the big takeaways is, um, That, you know, if you have a technology that is, like, really, really general and has billions of use cases and, like, ordinary people can use, like, launch it to, launch it to billions of people. I remember when I joined Google, there were, like, a lot of people working on this enterprise search appliance, which, you know, it was okay. Like, I think maybe somebody had this conventional wisdom that, like, Um, B to B is the only way to make money. But, like, what it actually turned out was, like, the much bigger thing was, uh, you know, was, uh, B to C, you know, like, just serve something to, serve something to everybody.

AI assessment note: “I'd say one of the big takeaways is, um, That, you know”

Q said there about kind of in the early days, you could download kind of the data of the internet, so to speak. Character is producing a ton of proprietary data within your Conversations. As are many verticalized solutions, be it in medical, be it in finance, be it wherever. To what extent is the value in proprietary, like, data ownership versus it will still and always be downloadable by everyone?

A The data that you get from users is great because It tells people like what, you know, what users like or like what users like in, in some particular, um, application. It's, it's kind of like a, you know, training a human. Like most of what's important is like you have, um, you know, decades of experience training your brain on stuff that is not really specific to your task at hand or your occupation, but you've kind of Gotten a generally good understanding of the world and gotten generally intelligent. And then you can, uh, improve on that dramatically by getting a smaller amount of training in the task you're doing. Both, both will contribute. And we, we do have like a huge amount of data flowing in from users, like just how, you know, obviously we're very, you know, very careful to not Compromise anyone's privacy, but just based on, you know, an aggregate, how, how people are, you know, using the service, we can learn to make it better.

AI assessment note: “Both, both will contribute.”

Q When you get up in front of the company, and you share, you, there's many people in the team now, what do you chest pump as the mission?

A I like this, Sort of motto of, you know, a billion users inventing a billion use cases, because that, that's sort of the superpower of this technology. And, you know, it sort of puts our company in, you know, in the right place of, we can't really, you know, guess, you know, what are the best uses, uses of this technology. And, you know, we've just You know, observe time and time again, like you put one thing out there and that's not really what people want. And somebody else out there, like find something better to do with it. We put up as an example, like a psychologist character, like maybe, maybe you want to, you know, talk to something and feel better, you know, and you know, that gets a little bit of use, but then what we hear a lot more from users is like, I'm talking to a video game character who's now my new therapist, and this makes me feel better. We had no idea that was, that was going to go on. Um, but like, and then, you know, there's this huge, like, use case in, like, some mix of, like, entertainment and companionship and emotional support. We were totally not experts in this stuff. Like, you know, like, our job is just to put out something general And, like, just respect the agency of, like, our users, of everybody out there to, uh, you know, to, to do what they want with this stuff.

AI assessment note: “motto of, you know, a billion users inventing a billion use cases”

Q When you think about the incredible growth that you've seen, four hundred and fifty million messages a day, twenty million users, what do you think have been the ones, two biggest elements that have driven that growth?

A Well, one is that we launched. Like, that's definitely been a frustration, you know, in the past things seem, uh, You know, potentially too much brand risk at larger companies to like actually launch and get it out there. Another aspect is, you know, we launched something general. We sort of let, let people find the use cases. And then the other is there are like massive needs out there in the world. Like, okay, there are billions of people who, you know, feel like they need someone to talk to. So, okay. Combine those elements. Provides people with something general that they can use, and there are people out there with needs, and, and they're going to find it.

AI assessment note: “One is that we launched... Another aspect is, you know, we launched something general.”

Q What do you think is the hardest product challenge you face today? It's a difficult product paradigm that you face. So many different use cases, so many different people's needs. What do you think of the hardest product paradigms for you as a team to face?

A The main things we need to do is, um, make it, make it very general. So it, you know, so we're not like cutting down on the use cases, make it usable. People think of those two things as being in opposition to each other, being versatile, And being usable. And like, you know, we talked to like some, uh, you know, potential product managers early on and they all say the same thing. Oh yeah. Pick your verticals, narrow it down to me, to make it usable. And like, no, we're not going to hire these people. That's like the opposite of what we want to do. We want to build something that is usable, but very, very general purpose. So there's, there's sort of that dichotomy.

AI assessment note: “People think of those two things as being in opposition to each other, being versatile, And being usable.”

Q Can I be, I'm fascinated to hear your thoughts. I, I totally agree with you in terms of the horizontal use case. I'm fascinated by all the different ways they talk to it. Do you not worry that we're losing touch with other humans? They've got no one to talk to, and so they talk to a machine.

A I think there's, like, you know, huge, like, Value in the connections between people and, you know, moral value as well. Like last thing I want to do is to, you know, take people away from, from, from human connection in a lot of ways. Um, you know, we want to help with, you know, human connection. A lot of the people who, who don't have friends and who, you know, are not. As well connected, one big, uh, source of that is just social anxiety and, you know, like, you know, there, yeah, there are huge numbers of people who are, like, uncomfortable, and we've got, you know, we've gotten testimonials of people who said that they were uncomfortable, you know, talking to other people, and, like, this is great practice. This is actually helped them, uh, build up practice in either social situations.

AI assessment note: “this is great practice. This is actually helped them, uh, build up practice”

Q was, like, super excitement in, like, a month period. But there wasn't this sustained belief that we have today in AI transforming the whole way society works. And I guess my question to you is, like, where we are today, is that the result of technological progress recently, very recently, or is it the result of investors in society catching up with what's been developing over a much longer period?

A I'd say, I'd say it's both. Like, uh, you know, I, I think there's been a lot of technological progress, uh, both quantitatively and qualitatively. The models that were there in like, 2016 or something were, you know, were too dumb to be fun. The, the neural models. Then there were, was this, all of the chat bot stuff you heard about back then was these rule-based systems that You know, we're just highly fragile and not going anywhere. You just needed more and more rules, and there's, like, no way to think of, like, all the things that could come up, and they just don't generalize. So that wasn't going to work, but at the same time, the, um, you know, we, we, we were progressing on, on the, on the neural network solutions, which, which we're going to scale. It took some amount of time. I'd say around 2020 was when Sort of, like, really impressive stuff was, was sort of in the lab, but not launched. So my co-founder, uh, Daniel De Freitas, like, he's, he's a very cool, smart, scrappy guy. He, he's, like, on this lifelong mission to do chatbots. You know, since he was a kid in Brazil, he's wanted to build, like, open domain chatbots.

AI assessment note: “I'd say, I'd say it's both. Like, uh, you know, I, I think”

Q was, like, super excitement in, like, a month period. But there wasn't this sustained belief that we have today in AI transforming the whole way society works. And I guess my question to you is, like, where we are today, is that the result of technological progress recently, very recently, or is it the result of investors in society catching up with what's been developing over a much longer period?

A I'd say, I'd say it's both. Like, uh, you know, I, I think there's been a lot of technological progress, uh, both quantitatively and qualitatively. The models that were there in like, 2016 or something were, you know, were too dumb to be fun. The, the neural models. Then there were, was this, all of the chat bot stuff you heard about back then was these rule-based systems that You know, we're just highly fragile and not going anywhere. You just needed more and more rules, and there's, like, no way to think of, like, all the things that could come up, and they just don't generalize. So that wasn't going to work, but at the same time, the, um, you know, we, we, we were progressing on, on the, on the neural network solutions, which, which we're going to scale. It took some amount of time. I'd say around 2020 was when Sort of, like, really impressive stuff was, was sort of in the lab, but not launched. So my co-founder, uh, Daniel De Freitas, like, he's, he's a very cool, smart, scrappy guy. He, he's, like, on this lifelong mission to do chatbots. You know, since he was a kid in Brazil, he's wanted to build, like, open domain chatbots.

AI assessment note: “I'd say, I'd say it's both.”

Q said there about kind of in the early days, you could download kind of the data of the internet, so to speak. Character is producing a ton of proprietary data within your Conversations. As are many verticalized solutions, be it in medical, be it in finance, be it wherever. To what extent is the value in proprietary, like, data ownership versus it will still and always be downloadable by everyone?

A The data that you get from users is great because It tells people like what, you know, what users like or like what users like in, in some particular, um, application. It's, it's kind of like a, you know, training a human. Like most of what's important is like you have, um, you know, decades of experience training your brain on stuff that is not really specific to your task at hand or your occupation, but you've kind of Gotten a generally good understanding of the world and gotten generally intelligent. And then you can, uh, improve on that dramatically by getting a smaller amount of training in the task you're doing. Both, both will contribute. And we, we do have like a huge amount of data flowing in from users, like just how, you know, obviously we're very, you know, very careful to not Compromise anyone's privacy, but just based on, you know, an aggregate, how, how people are, you know, using the service, we can learn to make it better.

AI assessment note: “Both, both will contribute. And we, we do have like a huge amount of data”

Q How did that change how you think? Go for bigger?

A Um, yeah, well, well, I, I think like right now I, I've, uh, you know, started this, uh, this company character and we're taking this large language model technology and, you know, we are just direct to consumer first. Like here is something that is even more, uh, more versatile and more easy to use than even web search, you know, in that like, okay, you can, Use it to be your friend or do your homework or like brainstorming or get ideas or like a billion things. We haven't even thought of the best use cases yet. And then it's massively usable usable. Like all you have to do is talk to it. So it has these two properties. And to me, that means go and like launch it to like everyone in the world and let everybody in the world use it where I think some, Some of the other, uh, you know, some of the other companies are taking a more, uh, a more like B to B sort of approach with, you know, like a, uh, you'll have a foundational model company and then like verticalized application companies on top of it. So I'm really, um, like inspired by the Google model of like full stack end to end all the way from like basic research to, um, You know, to launch a product directly to consumers. It's super fun. It's super motivating, you know, because like engineers like building stuff and then launching it and having like everyone use it, uh, immediately. And, uh, and then it also lets you do all …

AI assessment note: “to me, that means go and like launch it to like everyone in the world”

Q When you think about the incredible growth that you've seen, four hundred and fifty million messages a day, twenty million users, what do you think have been the ones, two biggest elements that have driven that growth?

A Well, one is that we launched. Like, that's definitely been a frustration, you know, in the past things seem, uh, You know, potentially too much brand risk at larger companies to like actually launch and get it out there. Another aspect is, you know, we launched something general. We sort of let, let people find the use cases. And then the other is there are like massive needs out there in the world. Like, okay, there are billions of people who, you know, feel like they need someone to talk to. So, okay. Combine those elements. Provides people with something general that they can use, and there are people out there with needs, and, and they're going to find it.

AI assessment note: “we launched something general. We sort of let, let people find the use cases.”

Q Can I be, I'm fascinated to hear your thoughts. I, I totally agree with you in terms of the horizontal use case. I'm fascinated by all the different ways they talk to it. Do you not worry that we're losing touch with other humans? They've got no one to talk to, and so they talk to a machine.

A I think there's, like, you know, huge, like, Value in the connections between people and, you know, moral value as well. Like last thing I want to do is to, you know, take people away from, from, from human connection in a lot of ways. Um, you know, we want to help with, you know, human connection. A lot of the people who, who don't have friends and who, you know, are not. As well connected, one big, uh, source of that is just social anxiety and, you know, like, you know, there, yeah, there are huge numbers of people who are, like, uncomfortable, and we've got, you know, we've gotten testimonials of people who said that they were uncomfortable, you know, talking to other people, and, like, this is great practice. This is actually helped them, uh, build up practice in either social situations.

AI assessment note: “This is actually helped them, uh, build up practice in either social situations.”

Q How did that change how you think? Go for bigger?

A Um, yeah, well, well, I, I think like right now I, I've, uh, you know, started this, uh, this company character and we're taking this large language model technology and, you know, we are just direct to consumer first. Like here is something that is even more, uh, more versatile and more easy to use than even web search, you know, in that like, okay, you can, Use it to be your friend or do your homework or like brainstorming or get ideas or like a billion things. We haven't even thought of the best use cases yet. And then it's massively usable usable. Like all you have to do is talk to it. So it has these two properties. And to me, that means go and like launch it to like everyone in the world and let everybody in the world use it where I think some, Some of the other, uh, you know, some of the other companies are taking a more, uh, a more like B to B sort of approach with, you know, like a, uh, you'll have a foundational model company and then like verticalized application companies on top of it. So I'm really, um, like inspired by the Google model of like full stack end to end all the way from like basic research to, um, You know, to launch a product directly to consumers. It's super fun. It's super motivating, you know, because like engineers like building stuff and then launching it and having like everyone use it, uh, immediately. And, uh, and then it also lets you do all …

AI assessment note: “that means go and like launch it to like everyone in the world”

Q I just don't get how you, I'm so I'm naive here. So I'm asking for education. But it's like, I'm trained on the thought that the more you specialize, the deeper, the richer conversation value you can provide. And so how do you provide quality high enough with such generalization?

A Yeah. I mean, that's been the magic of, uh, of, of neural language modeling. You know, the previous systems were all these rule-based systems that, you know, Like, fantastically complicated, you know, systems with millions of handwritten rules and, you know, like, uh, you know, really, really complicated. It required knowing something about linguistics and about state of mind and, like, all, all kinds of stuff. The new way of doing things with neural language models has none of that. Like, I could know, like, pretty much zero about language in particular. Um, other than it's like a sequence of words. So it has nothing to do with understanding language at all. And there are not millions of rules. It's, it's actually relatively simple, kind of like a big black box. It all boils down to this one beautiful, simple problem of you have this sequence of words. That's like the beginning of your document. Guess what the next word is. Like, give me odds on what the next word is in the sequence. And that, that problem is called, uh, language modeling. Just guess what the next word is based on the previous ones. You know, so I got involved with this, you know, around like, 2015, uh, that, you know, that, um, you know, there were some other folks at Google, uh, you know, working, you know, working on this problem. They're like, how, how good can we make it? And this struck me like, hey, thi…

AI assessment note: “that's been the magic of, uh, of, of neural language modeling.”

Q What are one to two of the biggest takeaways from your time? 20 years is a long time. How did it impact you?

A I'd say one of the big takeaways is, um, That, you know, if you have a technology that is, like, really, really general and has billions of use cases and, like, ordinary people can use, like, launch it to, launch it to billions of people. I remember when I joined Google, there were, like, a lot of people working on this enterprise search appliance, which, you know, it was okay. Like, I think maybe somebody had this conventional wisdom that, like, Um, B to B is the only way to make money. But, like, what it actually turned out was, like, the much bigger thing was, uh, you know, was, uh, B to C, you know, like, just serve something to, serve something to everybody.

AI assessment note: “I'd say one of the big takeaways is, um, That, you know, if you have a technology”

Q I just don't get how you, I'm so I'm naive here. So I'm asking for education. But it's like, I'm trained on the thought that the more you specialize, the deeper, the richer conversation value you can provide. And so how do you provide quality high enough with such generalization?

A Yeah. I mean, that's been the magic of, uh, of, of neural language modeling. You know, the previous systems were all these rule-based systems that, you know, Like, fantastically complicated, you know, systems with millions of handwritten rules and, you know, like, uh, you know, really, really complicated. It required knowing something about linguistics and about state of mind and, like, all, all kinds of stuff. The new way of doing things with neural language models has none of that. Like, I could know, like, pretty much zero about language in particular. Um, other than it's like a sequence of words. So it has nothing to do with understanding language at all. And there are not millions of rules. It's, it's actually relatively simple, kind of like a big black box. It all boils down to this one beautiful, simple problem of you have this sequence of words. That's like the beginning of your document. Guess what the next word is. Like, give me odds on what the next word is in the sequence. And that, that problem is called, uh, language modeling. Just guess what the next word is based on the previous ones. You know, so I got involved with this, you know, around like, 2015, uh, that, you know, that, um, you know, there were some other folks at Google, uh, you know, working, you know, working on this problem. They're like, how, how good can we make it? And this struck me like, hey, thi…

AI assessment note: “that's been the magic of, uh, of, of neural language modeling.”

Q What do you think is the hardest product challenge you face today? It's a difficult product paradigm that you face. So many different use cases, so many different people's needs. What do you think of the hardest product paradigms for you as a team to face?

A The main things we need to do is, um, make it, make it very general. So it, you know, so we're not like cutting down on the use cases, make it usable. People think of those two things as being in opposition to each other, being versatile, And being usable. And like, you know, we talked to like some, uh, you know, potential product managers early on and they all say the same thing. Oh yeah. Pick your verticals, narrow it down to me, to make it usable. And like, no, we're not going to hire these people. That's like the opposite of what we want to do. We want to build something that is usable, but very, very general purpose. So there's, there's sort of that dichotomy.

AI assessment note: “build something that is usable, but very, very general purpose”

Q the AI community? I wish there was a transparent log where you could see legitimacy of authors, and you could say AI publications or blogs, But I think so many quickly profess to be experts and actually just determining discovery wise, who is Yann LeCun? Who is Yoshua Bengio? Who is Noam Shazir versus who is someone who moved from web three and crypto and is now a AI expert?

A Uh, there's so much stuff being published. It's hard to, hard to know what's good. And I think a lot of that has to do with the fact that, you know, this field is kind of alchemy. Right now, like no one knows exactly what is going to work. So, you know, you have a lot of people trying lots and lots of different things and, you know, you, you can come up with hits by having like a good intuition of what will work on the ML side, kind of combined by like a good mathematical understanding of What will run fast on hardware that you can buy or that you can build? So there are some hits that come out and, you know, people will adopt and it's combined with a lot of noise. Um, like negative results are not useful because they could be negative because somebody just made a mistake and there could be, you know, a bug, you know, the, it didn't work for some other reason. What is interesting Is positive results that are proven out by experimentation. Like, you know, if somebody can say, hey, I did something and, you know, did better at this well-known problem, then that gets interesting. Then you, then everyone tries to figure out, okay, why does this work? How can I adopt it?

AI assessment note: “there's so much stuff being published. It's hard to, hard to know what's good.”

Q Can I ask, we asked about your joining Google. I think so many people are shaped by their past. When you think back on yours, How do you think about what you're running from?

A Yeah. Why did I start working on artificial intelligence? Like maybe, well, part, partially because it's just like fun and what I do for fun anyway. Like just what could be better than try to get the computer to do something that it currently can't do. But then, you know, the other thing is just to push technology forward there. You know, there are so many like technological, uh, you know, problems in the world that could be solved. You have like, fifty million people a year, like Like dying from stuff like old age and cancer and heart disease and like all kinds of stuff that we could potentially, uh, you know, potentially find cures for. So rather than directly working on say medical research or something, um, I think I've got a lot more leverage, like let's push AI technology and then, you know, that can help with, uh, a lot of the rest of it.

AI assessment note: “Why did I start working on artificial intelligence?”

Q I totally get you on the speed difference there. Do you think startups win then as a result in this next wave of AI innovation, entrepreneurship? Because a lot of people I have on the show now say, It's, I'm not, not less so on the Facebooks of the world, but more the Microsofts and the Adobe's win. Who wins startup or incumbent? If you have to pick a side.

A I'd say the users win. The users are going to have a lot, you know, a lot of options, but you know, on the, uh, on the business side, I think there can be, you know, a lot of winners. You know, there's going to be room for like, Multiple, multiple players in there, you know, big companies doing what big companies are good at startups doing what startups are good at. We're going to try to move our company from being a startup to being a big company as, you know, as fast as we can there. But a lot of just individuals and universities and such, you know, in that, like the hardware is progressing so fast that what you could do at a big company, you know, one year, a few years later, you're going to be able to do You know, at the university lab or in your garage.

AI assessment note: “I'd say the users win. The users are going to have a lot”

Q Can I ask, we asked about your joining Google. I think so many people are shaped by their past. When you think back on yours, How do you think about what you're running from?

A Yeah. Why did I start working on artificial intelligence? Like maybe, well, part, partially because it's just like fun and what I do for fun anyway. Like just what could be better than try to get the computer to do something that it currently can't do. But then, you know, the other thing is just to push technology forward there. You know, there are so many like technological, uh, you know, problems in the world that could be solved. You have like, fifty million people a year, like Like dying from stuff like old age and cancer and heart disease and like all kinds of stuff that we could potentially, uh, you know, potentially find cures for. So rather than directly working on say medical research or something, um, I think I've got a lot more leverage, like let's push AI technology and then, you know, that can help with, uh, a lot of the rest of it.

AI assessment note: “Why did I start working on artificial intelligence?”