The Exchanges

Q A lot of people suggest in the venture community, so you can poo poo this one, but like when they're denigrating kind of a lot of AI startups, they say, oh, it's a thin value layer on top of foundational models, and actually really the value accrues to the foundational model layer because it's this thin line on top. Is that fair? Or do you think that's total bullshit?

A You know, um, you asked me a very good question that often have a more subtle answer than, uh, than what would fit in a tweet. I think there are some very thin rapper companies out there that probably have very little moat, but there are also companies that, uh, people don't realize the complexity to make it really work. And they underestimate all of a sudden all the other stuff that Uh, companies need to get right to build a viable business. So concretely, uh, you know, you could think about Instagram, right? Instagram is like, what's the moat of Instagram? It's certainly not their AI and their backend and the brilliance of engineering, right? It's just like a fairly simple photo sharing app with some fun filters back in the day. None of that was rocket science. Turns out you can have moat other than, um, your, your backend AI model, right? It's distribution. It's partnerships. Uh, it's your sales funnel processes, uh, and so on. So there's a lot. And then there are also, um, areas where the default large language model will not do as well. So for instance, if you want it to be more factual, more up to date and have, uh, citations for the facts that it tells you, you need to have a search packet. And so at u.com, for instance, we've, We had to build this very complex search backend with a ton of data in it and knowing when to retrieve what facts from the internet so that your …

AI assessment note: “I think there are some very thin rapper companies out there that probably have very little moat, but”

Q Now, I said this to Kevin Scott at Microsoft, and he said, what's the moat with search? Go from Google to Bing, just the same, but you don't. You stay with Google. Well, he didn't obviously say that, because he's obviously with Bing. But my question to you then is like, how do you see the distribution of value across the LLM space with the recognition of that?

A That's why I said, if you're in that thin infrastructure layer, and that was an important qualification because OpenAI is a consumer app company. They make their revenue, the vast majority of their revenue from a consumer app called ChatGPT. If you're now just in that API infrastructure layer, it's very different. You have a lot more pressure. Anthropic has a lot more pressure to keep building the best models because Claude is so much smaller in terms of market share for the consumer app. And so that's why That analogy doesn't work. And you're right. Like consumers, once you're really famous and you cross that threshold of just like being well known, being the default for a lot of people, all the other LM apps companies are almost rounding errors to ChatGPT.

AI assessment note: “all the other LM apps companies are almost rounding errors to ChatGPT.”

Q Richard Socher, I'm fascinated to hear your thoughts on this. Obviously with you.com, you compete in many ways with Google on the search side. Richard, how do you think about this and Google's next steps?

A We don't see Google change and become a chat first search engine. They have some features somewhere else, but the main Google experience is the same. That big change will be hard for Google too, because they make five hundred million dollars a day With privacy invading advertisements on that page. And so you don't just really nilly change most of that page and you get rid of the five, six ads that are on top of that page, followed by a bunch of SEO and microsites that are not as good as the ad. So people click more on the ads. Like you don't just replace all of that with a chat, right? Cause you just lose hundreds of millions of dollars a day. And so there is still some innovators dilemma that will not make them change their main experience overnight.

AI assessment note: “We don't see Google change and become a chat first search engine.”

Q Richard, founder at You, you were chief data scientist at Salesforce before. I'm really intrigued. How do you feel about this lead position that OpenAI has that Douai just illustrated there, and what that means then for the potential for an open source competitor to rise?

A I mean, certainly you can't deny that OpenAI is ahead by a lot. I predicted that we'll have a GPT-IV equivalent model Before the end of the year, that's open source. Of course, GPD four keeps getting better and better. So my prediction was for the version we had like a few months ago, but I actually think that with models like Lama two from Facebook and everyone there, I do think. Open source will take over a lot of use cases, right? It's already getting close to GPD 3.5 when there's this much excitement and so many careers depending on understanding these models. Imagine all the researchers. In all these universities, right? They're all of a sudden kind of out of a job unless they have an LM that works really, really well and they can do useful things with it. They're not going to just say, oh, let's just from now on run our entire research agenda on some closed API that we cannot analyze and understand and improve and publish papers on. So they need to have a model to exist. And those are all very, very smart people now that don't have as many resources usually. They can't train a single model for like 20 or fifty million dollars, uh, cause they're in universities, but they're finding ways they are collaborating and they're probably going to work on foundational models that are fully open source. Uh, and we now see this with like surprisingly Facebook being at the very forefr…

AI assessment note: “I do think. Open source will take over a lot of use cases”

Q Richard Socher, I'm fascinated to hear your thoughts on this. Obviously with you.com, you compete in many ways with Google on the search side. Richard, how do you think about this and Google's next steps?

A We don't see Google change and become a chat first search engine. They have some features somewhere else, but the main Google experience is the same. That big change will be hard for Google too, because they make five hundred million dollars a day With privacy invading advertisements on that page. And so you don't just really nilly change most of that page and you get rid of the five, six ads that are on top of that page, followed by a bunch of SEO and microsites that are not as good as the ad. So people click more on the ads. Like you don't just replace all of that with a chat, right? Cause you just lose hundreds of millions of dollars a day. And so there is still some innovators dilemma that will not make them change their main experience overnight.

AI assessment note: “We don't see Google change and become a chat first search engine.”

Q Where do you think value most accrues if you were to bet on one in the next five years? Does the next wave of AI create more value for incumbents or more value for startups?

A I think it'll be a mix. I think there are companies that have been fully activated. I see like Salesforce having launched a bunch of really incredible features and made incredible announcements in their AI day that will make it harder for AI for service automation, for instance, because it's so core to their business. We've also seen Bing and Google copy what we have launched late last year with you chat. They've copied us in the sense that we've launched it earlier and then they launched something very similar three to four months after. At the same time, we don't see Google change and become a chat first search engine. They have some features somewhere else, but the main Google experience is the same. That kind of big change will be hard for Google too, because they make five hundred million dollars a day with privacy invading advertisements on that page. And so you don't just willy nilly change most of that page and you get rid of the five, six ads that are on top of that page, followed by a bunch of SEO and microsites that are not as good as the ads. So people click more on the ads. Like you don't just replace all of that with a chat, right? Cause you just lose hundreds of millions of dollars a day. And so there is still some innovators dilemma that will not make them change their main experience overnight. It's been very carefully tuned. Every shade of blue has been AB tes…

AI assessment note: “I think it'll be a mix. I think there are companies that have been fully activated.”

Q A lot of people suggest in the venture community, so you can poo poo this one, but like when they're denigrating kind of a lot of AI startups, they say, oh, it's a thin value layer on top of foundational models, and actually really the value accrues to the foundational model layer because it's this thin line on top. Is that fair? Or do you think that's total bullshit?

A You know, um, you asked me a very good question that often have a more subtle answer than, uh, than what would fit in a tweet. I think there are some very thin rapper companies out there that probably have very little moat, but there are also companies that, uh, people don't realize the complexity to make it really work. And they underestimate all of a sudden all the other stuff that Uh, companies need to get right to build a viable business. So concretely, uh, you know, you could think about Instagram, right? Instagram is like, what's the moat of Instagram? It's certainly not their AI and their backend and the brilliance of engineering, right? It's just like a fairly simple photo sharing app with some fun filters back in the day. None of that was rocket science. Turns out you can have moat other than, um, your, your backend AI model, right? It's distribution. It's partnerships. Uh, it's your sales funnel processes, uh, and so on. So there's a lot. And then there are also, um, areas where the default large language model will not do as well. So for instance, if you want it to be more factual, more up to date and have, uh, citations for the facts that it tells you, you need to have a search packet. And so at u.com, for instance, we've, We had to build this very complex search backend with a ton of data in it and knowing when to retrieve what facts from the internet so that your …

AI assessment note: “there are some very thin rapper companies out there... but there are also companies”

Q ah, incumbents don't move fast enough, and that's why startups win. And then you look at incumbents, and they're moving pretty fucking fast right now, and I'm going, oh, that doesn't hold up. Where do you think value most accrues if you were to bet on one in the next five years? Does it, does the next wave of AI create more value for incumbents or more value for startups?

A I think it'll be a mix. Uh, I think there are companies that have been fully activated. Um, you know, I, I see like Salesforce having launched a bunch of really incredible features and made incredible announcements in their AI day. Um, and that will be, that will make it harder for AI for service automation, for instance, cause it's so core to their business. Uh, we've also seen, uh, Bing and Google copy, you know, what we have launched last late last year with you chat. Uh, and you know, they may have worked on it before, like some people, you know, claim, oh, for, I mean, we've also met the prompt engineering in 2018, but like, they've copied us in the sense that, you know, we've launched it earlier, and then they launched something very similar, uh, three to four months after. At the same time, we don't see Google change and become a chat first search engine, right? They have some features somewhere else. But the main Google experience is the same. And so if you want to have a chat first search engine. That's like you.com is, is the only solution that has all these features of a search engine. It tells you the weather and it has stock predict like stock tickers and all of these other things, but it's chat first. That kind of big change will be hard for Google too, because they make five hundred million dollars a day with privacy invading advertisements on that page. Right. A…

AI assessment note: “I think it'll be a mix.”

Q Can I ask, you mentioned before, uh, something I thought was really interesting, and I want to start there because it leads to Search and you, but it's, you've been on a quest for a single pre-trained model. I just want to make sure everyone follows along with us in this conversation. For those that don't know, what do we have today, and why is that maybe inefficient?

A I mean, today it's finally changing. We've had, we've made progress towards that single model, but just, uh, even last year, um, or two or three years ago, The prevailing idea was that every task in natural language processing should have its own model. You have a sentiment analysis model that just classifies tweets as positive or negative. Then you have a summarization model that takes in some long input and then summarizes it in a few in fewer sentences. Uh, you have a translation model that just translates German to English. You have A question answering model that takes a context and says, who's the president, uh, in this Wikipedia article, uh, that's mentioned, and then you just give that. So there are all these different sub models, um, that people have worked on and some people have built their entire careers on just sentiment analysis models. Right. And so the difference in something I've been very excited about for, for pretty much a decade now, uh, is to have a model that you keep making better. That you keep adding to rather than restarting every new training run and so on. Kind of like imagine Wikipedia and everyone just keeps adding to Wikipedia and keeps making one dictionary better rather than everyone who wants to build a dictionary just starts their own dictionary company and then builds it from scratch. It just doesn't make as much sense. It makes sense when h…

AI assessment note: “The prevailing idea was that every task in natural language processing should have its own model.”

Q ah, incumbents don't move fast enough, and that's why startups win. And then you look at incumbents, and they're moving pretty fucking fast right now, and I'm going, oh, that doesn't hold up. Where do you think value most accrues if you were to bet on one in the next five years? Does it, does the next wave of AI create more value for incumbents or more value for startups?

A I think it'll be a mix. Uh, I think there are companies that have been fully activated. Um, you know, I, I see like Salesforce having launched a bunch of really incredible features and made incredible announcements in their AI day. Um, and that will be, that will make it harder for AI for service automation, for instance, cause it's so core to their business. Uh, we've also seen, uh, Bing and Google copy, you know, what we have launched last late last year with you chat. Uh, and you know, they may have worked on it before, like some people, you know, claim, oh, for, I mean, we've also met the prompt engineering in 2018, but like, they've copied us in the sense that, you know, we've launched it earlier, and then they launched something very similar, uh, three to four months after. At the same time, we don't see Google change and become a chat first search engine, right? They have some features somewhere else. But the main Google experience is the same. And so if you want to have a chat first search engine. That's like you.com is, is the only solution that has all these features of a search engine. It tells you the weather and it has stock predict like stock tickers and all of these other things, but it's chat first. That kind of big change will be hard for Google too, because they make five hundred million dollars a day with privacy invading advertisements on that page. Right. A…

AI assessment note: “I think it'll be a mix.”

Q I would love to start with top level because it is very noisy and it is very confusing to understand what's going on. When you evaluate where we are in the LLM landscape today, how do you evaluate the current status of where we are?

A Boy, I think AI is kind of this, this tide that's rising, uh, but on top of that tide, you have a lot of little hype bubbles that come up and down. And, you know, in some ways you can think about When Sam, for instance, says the next generation of models will be as good as a PhD student. The corollary there is that most jobs don't even require a PhD. If you do service for DoorDash or something, like you don't need a PhD to answer service questions. And so LMs are already good enough. They just need to be brought into companies to be actually made useful. And so I think that's sort of one state. And then the future state is, of course, we will get even better at reasoning. And at some point there were very, very narrow niches where these models can be as good or better than an expert human. Um, so there's still a lot of room to grow. And so in fact, I, it's such a confusing state that part of this book I'm writing, um, includes a chapter on what I call the upper bounds of intelligence, where I essentially group intelligence into 10 different dimensions, uh, or groups of dimensions. And then we can kind of say in this dimension, There is like a fairly low upper bound and we're fairly close to it. For example, object detection and computer vision. It's actually kind of solved. We can classify most objects on the planet. Now in the upper bound that that type of intelligence can eve…

AI assessment note: “AI is kind of this, this tide that's rising, uh, but on top”

Q Uh, and then why would you, what's the bull case to be excited about humanoids then? Cause I just listened to you there and I'm like, it's all completely rational, right and fair. Paint the bull case.

A I think the bull case is at home. Like when it actually, the speed doesn't matter, but there's just a whole host of many different tasks. So you can have a Roomba and the Roomba will do one thing better than a humanoid. You can have a dishwasher and it will do that thing better than a humanoid. But if you now also have like, 50 other things, sort my socks, and do this, and open the door, and get a package from outside, put it inside, and like 50 or a hundred other little tasks, none of which you actually have to, at scale, massively every day, all the time, so the custom robot form factor doesn't make sense, then I think humanoids can be helpful, and if they're quiet enough, and the task execution is quiet enough, you can have them work slowly all night, right? You have a party, And you wake up, and the whole place is clean again. Now, the problem is that that is, and it's like these cluttered environments that are all different. There's no standardization. That's really, really hard for AI.

AI assessment note: “I think the bull case is at home.”

Q I would love to start with top level because it is very noisy and it is very confusing to understand what's going on. When you evaluate where we are in the LLM landscape today, how do you evaluate the current status of where we are?

A Boy, I think AI is kind of this, this tide that's rising, uh, but on top of that tide, you have a lot of little hype bubbles that come up and down. And, you know, in some ways you can think about When Sam, for instance, says the next generation of models will be as good as a PhD student. The corollary there is that most jobs don't even require a PhD. If you do service for DoorDash or something, like you don't need a PhD to answer service questions. And so LMs are already good enough. They just need to be brought into companies to be actually made useful. And so I think that's sort of one state. And then the future state is, of course, we will get even better at reasoning. And at some point there were very, very narrow niches where these models can be as good or better than an expert human. Um, so there's still a lot of room to grow. And so in fact, I, it's such a confusing state that part of this book I'm writing, um, includes a chapter on what I call the upper bounds of intelligence, where I essentially group intelligence into 10 different dimensions, uh, or groups of dimensions. And then we can kind of say in this dimension, There is like a fairly low upper bound and we're fairly close to it. For example, object detection and computer vision. It's actually kind of solved. We can classify most objects on the planet. Now in the upper bound that that type of intelligence can eve…

AI assessment note: “LLMs are already good enough. They just need to be brought into companies”

Q One thing I worry about is, like, the excitement around robotics. I find that robotics have not had their ChatGPT moment on the foundation model side. How do you think about robotics as having their ChatGPT moment or lack of yet, and maybe excitement or lack of excitement that you have towards it moving forward?

A That is a great question. I think the tricky bit in robotics is that part of why ChatGPT had this amazing moment is that it's so general, right? You can just ask it anything. So the equivalent to the generality of a ChatGPT is a humanoid. But the humanoids don't work really well yet. The problem is when robotics, like the cost, right, you only want to build specific types of robots when there's a highly scalable process. And now once there's a highly scalable process, the humanoid form factor is not the most optimal factor, right? Like if you have like a highly scalable process is in the fields and agriculture, you don't want a bunch of humanoid robots hunching over and like weeding. No, you just get a frigging like Massive tractor with a bunch of lasers and, like, thousands of little arms and spray, like, cannons, and then you just, like, either zap the, the weeds away, you don't have them hand plucked by, like, a humanoid robot, right? And so, for almost every process that is highly repeatable, there's a better, more quickly evolved new hardware form than five fingers on two arms.

AI assessment note: “the tricky bit in robotics is that part of why ChatGPT had this amazing moment”

Q What did you believe that you now no longer believe?

A Like I said, I think a big one for me was this sort of skepticism about the future. I'll tell you a story. Like one of the co-founders of OpenAI and I started a bet, I think seven years ago or so, where he said, we'll have AGI in like nine or 10 years. And I was like, I mean, I'll work hard. On research to make my prediction be wrong, but I really don't think we will. This was at, like, an AI conference, and we're both, like, still more, more junior than we are now, and he had this, like, really strong belief, and, you know, they worked on robotic hands, and they're like, another step towards AGI, and I'm like, I mean, it was a cool robotics project. They worked on Dota game playing at OpenAI, and I'm like, that's a cool RL project, and they're like, another step towards AGI, and then, you know, they saw, like, our prompt engineering paper, which they cited, and, like, like, that NLP route was a More legit step towards AGI than, than the previous things. But long story short, we, we did this bet and in the bet, uh, he has to win. I think it ends in 20, 27. So three things have to be true. We have to have a personal robot that cleans the whole house the way my cleaning team does. And it needs to be purchasable, like for reasonable amounts of money. It needs to solve a millennium math problem. And it needs to like translate a book perfectly so that the actual author would be like…

AI assessment note: “a big one for me was this sort of skepticism about the future.”

Q You said there about kind of reasonable pricing with bio. Do you think we're in a bubble in terms of AI early stage?

A Again, this is sort of where I think the analogy is the tide is rising, but it's also hot. So from time to time there are bubbles on top I don't think we're in an AI bubble period. I think intelligence, the fact that the marginal cost of intelligence goes down is the same as like the marginal cost of electricity or coal or something going down, but, but we will just use it more and more. Like a couple of years ago, I, I tweeted and read about this thing called Jevons paradox. A couple of like weeks or months ago, some other people have found it also and talk about it a lot, but it is for those who haven't yet seen it, it's a very useful analogy here. So in, uh, the, First industrial revolution, like, 18 sixties or so, Jevons was an economist, and he looked into the price of coal. And a lot of the smartest engineers and mines at the time made more and more efficient coal, like steam engines. And so he eventually thought, and a lot of people thought, well, if steam engines get more and more efficient, then we'll need less and less coal, so the price of coal will go down. But what actually happened is you just use steam engines in more and more places. Eventually, you know, they could create electricity, they can move Uh, steamboats and so on. And you just used it more and more. And so the price of coal actually went up instead. And so I think the analogy here is that yes, AI, lik…

AI assessment note: “I don't think we're in an AI bubble period.”

Q And so it's like, actually, is that very valuable? Does that not prove that the ultimate value in this market is consumer brand?

A In some ways, I think AI has, has gotten so exciting for so many people that the startup world is going back to the basics. Like, you know, it's just like when there were a hundred photo sharing apps and only one Instagram and maybe a Flickr and so on, uh, no pun intended, Flickr, like, I think we'll see something similar in AI. It just goes back to is your branding good, your marketing, your sales, your distribution, and then, of course, a lot of the technology things, but those get commoditized more and more, just like sharing photos was a fairly commodity capability, but a lot of little subtle details were better for Instagram. Now that's consumer. In consumer, we usually end up in monopoly or duopoly situations. Enterprise is a very different world.

AI assessment note: “It just goes back to is your branding good, your marketing, your sales”

Q Richard, founder at You, you were chief data scientist at Salesforce before. I'm really intrigued. How do you feel about this lead position that OpenAI has that Douai just illustrated there, and what that means then for the potential for an open source competitor to rise?

A I mean, certainly you can't deny that OpenAI is ahead by a lot. I predicted that we'll have a GPT-IV equivalent model Before the end of the year, that's open source. Of course, GPD four keeps getting better and better. So my prediction was for the version we had like a few months ago, but I actually think that with models like Lama two from Facebook and everyone there, I do think. Open source will take over a lot of use cases, right? It's already getting close to GPD 3.5 when there's this much excitement and so many careers depending on understanding these models. Imagine all the researchers. In all these universities, right? They're all of a sudden kind of out of a job unless they have an LM that works really, really well and they can do useful things with it. They're not going to just say, oh, let's just from now on run our entire research agenda on some closed API that we cannot analyze and understand and improve and publish papers on. So they need to have a model to exist. And those are all very, very smart people now that don't have as many resources usually. They can't train a single model for like 20 or fifty million dollars, uh, cause they're in universities, but they're finding ways they are collaborating and they're probably going to work on foundational models that are fully open source. Uh, and we now see this with like surprisingly Facebook being at the very forefr…

AI assessment note: “certainly you can't deny that OpenAI is ahead by a lot”

Q Where do you think value most accrues if you were to bet on one in the next five years? Does the next wave of AI create more value for incumbents or more value for startups?

A I think it'll be a mix. I think there are companies that have been fully activated. I see like Salesforce having launched a bunch of really incredible features and made incredible announcements in their AI day that will make it harder for AI for service automation, for instance, because it's so core to their business. We've also seen Bing and Google copy what we have launched late last year with you chat. They've copied us in the sense that we've launched it earlier and then they launched something very similar three to four months after. At the same time, we don't see Google change and become a chat first search engine. They have some features somewhere else, but the main Google experience is the same. That kind of big change will be hard for Google too, because they make five hundred million dollars a day with privacy invading advertisements on that page. And so you don't just willy nilly change most of that page and you get rid of the five, six ads that are on top of that page, followed by a bunch of SEO and microsites that are not as good as the ads. So people click more on the ads. Like you don't just replace all of that with a chat, right? Cause you just lose hundreds of millions of dollars a day. And so there is still some innovators dilemma that will not make them change their main experience overnight. It's been very carefully tuned. Every shade of blue has been AB tes…

AI assessment note: “I think it'll be a mix.”

Q Speaking of not having one single model to rule them all, we did an incredible discussion with Richard Soch at you.com. I've had guests on the show before, and they say the model size isn't so important, but it's the data size that is. Is that wrong?

A It's totally not wrong. It's just not mutually exclusive. You need a large model and you need A lot of training data for that model, either of them in isolation, like, you know, just like imagine the simplest neural network will just predict a single one dimensional output line, right? That's like a regression analysis. You have some input X, some output Y, and you try to model where it goes. You can model that with a handful of neurons. And the simplest one is just like a line, right? A linear regression. And that model has even fewer parameters. Long story short, if you now give this linear regression model Billions and billions of training data, it's not gonna learn magically anything but a simple linear line. But if you give the model billions and billions of parameters, it can learn all kinds of very complex predictive functions and abilities. So concretely, the big breakthrough on top of this idea of prompt engineering of being able to have a single model was to also use language modeling as one of those tasks. And the idea of language modeling is you just predict the next word, which Is very easy to get a lot of data for because you can use anything on the internet and so on, but it's actually incredibly hard and to do it really, really well, you have to learn so much about the world, right? If I'm just, I have the sentence, like I'm in New York city and I'm driving nort…

AI assessment note: “It's totally not wrong. It's just not mutually exclusive. You need a large model”

Q You are very kind. I, I appreciate all ego inflation, but I'd love to start today with a little bit of context, because you've been in the world of ML and NLP for a long time. How did you make your first forays into the world of ML and NLP, and what did that look like?

A Boy, that goes back to, uh, to 20 oh three. 2003 is when I started linguistic computer science at Leipzig university, kind of the forays and the early days of natural language processing. But then I actually felt like there wasn't enough math in it. Uh, and I switched to computer vision and, uh, did my masters in medical computer vision, uh, and, and a lot more sort of statistical and pattern recognition. Recognition. Statistical learning. And then during my PhD, I saw, uh, some folks work in deep learning and neural networks for very small, uh, images, you know, 32 by 32 pixel images of digits and things like that. And I thought, couldn't we use these ideas that they're using in vision for natural language processing? And that sort of started, uh, down, uh, the road of that eventually that took a lot of and Uh, contextual vectors and inventing prompt engineering and, and trying to train a single model for all of NLP.

AI assessment note: “2003 is when I started linguistic computer science at Leipzig university”

Q I've had guests on the show before and they say the model size isn't so important, but it's the data size that is. Is that wrong?

A It's, it's totally not wrong. It's just not mutually exclusive. You need a large model and you need a lot of training data for that model. Either of them in isolation, like, you know, just like imagine if the simplest neural network will just predict a single one dimensional output line, right? That's like a regression analysis. You have some input X. Some output Y and you try to model sort of where it goes. That's, you know, you can model that with a handful of neurons. Um, and the simplest one is just like a line, right? A linear regression. Uh, and that, that model has even fewer parameters and like two in a, you know, two dimensional kind of, uh, input, uh, or, uh, space. And so, uh, long story short, if you now give this linear regression model, Billions and billions of training data. It's not gonna learn magically anything but a simple linear line. But if you give the model billions and billions of parameters, it can learn all kinds of very complex predictive, uh, functions and abilities. So concretely, um, the big breakthrough, uh, on top of this idea of prompt engineering, of being able to have a single model was to also use language modeling as One of those tasks. Uh, it was actually on our to-do list, uh, but we didn't get to it before others did, um, after we published, uh, the Deca NLP paper. And the idea of language modeling is you just predict the next word, which…

AI assessment note: “It's, it's totally not wrong. It's just not mutually exclusive.”

Q Can I ask, you mentioned before, uh, something I thought was really interesting, and I want to start there because it leads to Search and you, but it's, you've been on a quest for a single pre-trained model. I just want to make sure everyone follows along with us in this conversation. For those that don't know, what do we have today, and why is that maybe inefficient?

A I mean, today it's finally changing. We've had, we've made progress towards that single model, but just, uh, even last year, um, or two or three years ago, The prevailing idea was that every task in natural language processing should have its own model. You have a sentiment analysis model that just classifies tweets as positive or negative. Then you have a summarization model that takes in some long input and then summarizes it in a few in fewer sentences. Uh, you have a translation model that just translates German to English. You have A question answering model that takes a context and says, who's the president, uh, in this Wikipedia article, uh, that's mentioned, and then you just give that. So there are all these different sub models, um, that people have worked on and some people have built their entire careers on just sentiment analysis models. Right. And so the difference in something I've been very excited about for, for pretty much a decade now, uh, is to have a model that you keep making better. That you keep adding to rather than restarting every new training run and so on. Kind of like imagine Wikipedia and everyone just keeps adding to Wikipedia and keeps making one dictionary better rather than everyone who wants to build a dictionary just starts their own dictionary company and then builds it from scratch. It just doesn't make as much sense. It makes sense when h…

AI assessment note: “every task in natural language processing should have its own model.”

Q I had Alex, sorry, I'm, I'm like taking a collection of wisdom from other people and throwing it at you to hear your thoughts. I had Alex Nabla on the show, uh, and he said that actually a company's ability to transition between model is what will determine their success moving forwards. How do you think about companies transitioning between models as a differentiator and advantage mechanism?

A You can almost start to, in, in the future, think about, uh, The LMs as, like, akin in some ways to a database. You don't really care about which database people are using nowadays, right? You, if you want to go really big, you might use an Oracle database. If you just want to build a smaller thing, use some MySQL open source database, and there's a bunch of others in between. And I think it won't matter that much, uh, which database you use, just like it won't matter that much, which LM you use, but it matters what you do with it, how you tune it, uh, what kind of training data you add onto it to, to fine tune it. Uh, how you retrieve facts into it so it can reason well over all of those things. How you may be prompted to run multiple chain of thought steps to get to a conclusion. All of those things I think will matter more and more. And so I would argue that it's even more important, um, than switching between models is to be able to incorporate multiple different models. Cause you may have a predictive forecasting model and that It's important to include into an LM. Like an LM won't be as good in doing a financial forecast because that's not what they're trained on. Like LM's help a ton for all things, natural language, because they understand so much about natural language and will have so much world knowledge, but that doesn't help you if you just have a huge sequence of …

AI assessment note: “even more important, um, than switching between models is to be able to incorporate multiple”

Q you and it provides a great answer. I now don't go to the original site because you provided it for me instead. That site, often a news site or an information site or a content site, relies on clicks and traffic. How does the next generation business model of the internet work when that attribution and that throughput is not to the provider, but it is to this chat search?

A Yeah. Yeah. It's a great question. Um, our answer to this is that we need to have that search engine that we've built be an open platform, um, where you can actually contribute your apps to and your content to, uh, and then if it actually brings up the context from your app. Uh, and we make money with it, then you can also, you know, participate in making that money. Or you can have like, let's say, um, you're, you have some subscription model, right? And you have some data that you show publicly, but then you have to subscribe. You can have that subscription happening right within your search engine by virtue of this being a much more open platform than Google. Now we've launched this open platform last year, but to be honest, we haven't had A ton of really amazing apps being added to this platform because we just don't have hundreds of millions of users, and so if your, you know, app, like, lets you book a kayaking trip in Indonesia or something, you know, by the end, like, you, you only get, like, five people, you know, it, it's been slow in, uh, an uptake on that open platform, but I think that is philosophically the right solution, and I hope people will start collaborating with us, because if we don't win and others win, then they'll be actually fully left out.

AI assessment note: “our answer to this is that we need to have that search engine”

Q wonderful diplomat. Uh, so I totally understand that. I do want to discuss one final element though, and we, we touched on it before the show actually, but it is around AGI because there's a lot of people who are very excited and optimistic about AGI, but you mentioned to me that people might be overly optimistic. Why do you think that people are potentially overly optimistic around AGI, Richard?

A I think, um, I think it's very calm, like, uh, natural for people to look at, uh, a, Type of progress and then extrapolate it further and further. You know, you look at a lot of historical analogies, like mechanics made a lot of, uh, progress, right? And some famous Kings had like little, you know, dolls that could play songs and drum and stuff like that. And they're like, oh, the brain is a bunch of cogs moving around too. And like everything is mechanical in the world. And so we'll build these robots and, you know, the best they could do was the actual mechanical Turk where someone sits below a table and pretends to move the robot, uh, and, and play chess or something like that. I think there's a little bit of, of, uh, some, some overly strong optimism, um, because we have, again, made a ton of progress, right? Like it's undeniable how much better AI has, has gotten. Just like with flight, for instance, human flight, we went from, uh, The first motorized human flight. Uh, and then literally. 3040 years later, we could fly loopings with machine guns and full metal airplanes high up on the speed of sound. And you're just like, wow, I mean, at this rate of progress, we're going to have vacations on the moon and we're going to have flying cars and like everyone will just fly everywhere all the time and, and so on. And then in the fifties, the whole thing just. Stopped and we're f…

AI assessment note: “natural for people to look at a type of progress and then extrapolate it further”

Q I had Alex, sorry, I'm, I'm like taking a collection of wisdom from other people and throwing it at you to hear your thoughts. I had Alex Nabla on the show, uh, and he said that actually a company's ability to transition between model is what will determine their success moving forwards. How do you think about companies transitioning between models as a differentiator and advantage mechanism?

A You can almost start to, in, in the future, think about, uh, The LMs as, like, akin in some ways to a database. You don't really care about which database people are using nowadays, right? You, if you want to go really big, you might use an Oracle database. If you just want to build a smaller thing, use some MySQL open source database, and there's a bunch of others in between. And I think it won't matter that much, uh, which database you use, just like it won't matter that much, which LM you use, but it matters what you do with it, how you tune it, uh, what kind of training data you add onto it to, to fine tune it. Uh, how you retrieve facts into it so it can reason well over all of those things. How you may be prompted to run multiple chain of thought steps to get to a conclusion. All of those things I think will matter more and more. And so I would argue that it's even more important, um, than switching between models is to be able to incorporate multiple different models. Cause you may have a predictive forecasting model and that It's important to include into an LM. Like an LM won't be as good in doing a financial forecast because that's not what they're trained on. Like LM's help a ton for all things, natural language, because they understand so much about natural language and will have so much world knowledge, but that doesn't help you if you just have a huge sequence of …

AI assessment note: “I would argue that it's even more important, um, than switching between models”

Q Can I ask a weird one? Does the AI bubble that we see today, specifically in funding and interest, does it help you as an OG, or does it hurt you more? It's an increased competition for talent, it means there's a lot of money flying around, often cases inefficient, but then it does lead to innovation. Does it help or hurt more, do you think?

A I mean, overall it's, it's great, uh, to have a lot of funding. It means, you know, there, there's a lot of, a lot more, uh, fuel in, in this fire. And I think we can see, um, the result of that, right? There are a lot of companies that are pushing hard in a lot of different directions, uh, in the space. And I think overall the pie is just growing and, uh, it is funny to think back at two, 2010, 2011, uh, when we had the deep learning workshop. Uh, cause most of the rest of the AI community didn't care about neural networks or any of that. And that little workshop was 40 people or so. And, you know, Jeff and Yann LeCou and Yosha Bengio, uh, entering, uh, you know, me and like a, like a few dozen other like PhD students were all just like hanging out and, and then chatting about how this technology could be useful. And now it's like all of the AI conferences are neural network conferences. Um, and, and it's just that, it's incredible to see that impact and, and to have been there, uh, and in that community for so long.

AI assessment note: “overall it's, it's great, uh, to have a lot of funding.”

Q wonderful diplomat. Uh, so I totally understand that. I do want to discuss one final element though, and we, we touched on it before the show actually, but it is around AGI because there's a lot of people who are very excited and optimistic about AGI, but you mentioned to me that people might be overly optimistic. Why do you think that people are potentially overly optimistic around AGI, Richard?

A I think, um, I think it's very calm, like, uh, natural for people to look at, uh, a, Type of progress and then extrapolate it further and further. You know, you look at a lot of historical analogies, like mechanics made a lot of, uh, progress, right? And some famous Kings had like little, you know, dolls that could play songs and drum and stuff like that. And they're like, oh, the brain is a bunch of cogs moving around too. And like everything is mechanical in the world. And so we'll build these robots and, you know, the best they could do was the actual mechanical Turk where someone sits below a table and pretends to move the robot, uh, and, and play chess or something like that. I think there's a little bit of, of, uh, some, some overly strong optimism, um, because we have, again, made a ton of progress, right? Like it's undeniable how much better AI has, has gotten. Just like with flight, for instance, human flight, we went from, uh, The first motorized human flight. Uh, and then literally. 3040 years later, we could fly loopings with machine guns and full metal airplanes high up on the speed of sound. And you're just like, wow, I mean, at this rate of progress, we're going to have vacations on the moon and we're going to have flying cars and like everyone will just fly everywhere all the time and, and so on. And then in the fifties, the whole thing just. Stopped and we're f…

AI assessment note: “natural for people to look at, uh, a, Type of progress and then extrapolate”

Q warning that this is a shit question, ok, so warning ahead of time. Are you concerned by the job displacement question, and just the awareness that when you look at industrial revolutions and technological revolutions, there's always decade-long actually transition periods, whereas here it seems like the transition period is years, like a couple of years, not decades. Are you concerned, or do you think we're over worrying about this?

A A hundred percent. I do think, you know, past industrial revolutions, um, you know, they usually happen for an individual at a surprising rate, right? When you're a weaver, um, and you got some big machine now, and it just like totally automates making clothes. Like, you will hate that machine, right? And the Luddites tried to destroy those machines, and, and there are people who want to slow down and destroy certain kinds of AI that, that change their, their jobs. And it's a hundred percent understandable, and, and I feel empathy, uh, and sympathy with, with those people. I do think it would be nice to have social systems that catch some of those, uh, people and help them learn new kinds of skills, help them incorporate, Those new technologies into their workflows so that, you know, if you're an illustrator and you used to be able to charge a thousand dollars for an illustration, cause it took you three days. Now it takes you three minutes, maybe 30 minutes. If you're really trying to like very carefully, like prompt engineer something, um, or maybe create a style, it takes you a few days to create a style and then you can create the next million images in like three minutes in that new style that you input into the model. If you don't use it and you still expect to get paid for three days, but now they're like, Thousands of people can do it in three minutes. Like that, you kn…

AI assessment note: “A hundred percent. I do think, you know, past industrial revolutions”