The Exchanges

Q compression because they don't charge more per seat, but they have AI infused in all of their product, and so you can create, you know, anything with AI in their products, and obviously each query costs money. And so it's costing them more money and they're making the same revenue. Will we actually be able to make more revenue per user or will it just create a better customer experience?

A There's two different camps right now. Some people are pricing the exact same with AI features and using it to drive expansion in their business. And then the other folks like Microsoft, like Salesforce, like, uh, like Notion as well, um, they're charging for the AI features and getting a bigger business as a product. Both of those strategies are, are fine and super reasonable. For folks like Canva who are keeping the same price, um, I mean, I think it's a good bet. They want to grow their user base. They want to expand their user set. Just give them the most useful product possible. At the moment, don't worry about margins because the cost of AI is falling super, super quickly. Um, I think that's reasonable.

AI assessment note: “There's two different camps right now. Some people are pricing the exact same”

Q Does that continue in that same progressive advancement? And what I mean by that is, do we continue to see that same scaling advantages, or does it actually plateau at some point? As you said there, you know, we always hear about Moore's Law. At some point, It just becomes a better calculator for the iPhone.

A Yeah, I mean, I think it certainly requires exponential input. You know, you need to continuously be doubling your compute in order to sustain linear gains in, in intelligence. Um, but I, I think that probably goes on for a very, very, very long time. Um, it'll just keep getting smarter, but you run into, like, economic constraints, right? Uh, not a lot of people bought the original GPT-IV. Certainly not a lot of enterprises, because it was huge. It was massive. Super inefficient to serve. So costly. Not smart enough to justify that cost. And so I think there's a lot of pressure on making smaller, more efficient models. Smarter via data and algorithms, methods, rather than just scaling up due to market forces, just pressure on price.

AI assessment note: “I think that probably goes on for a very, very, very long time.”

Q Yeah. And so you were expecting it to pop off relatively quickly, I take it?

A No, not, not at that moment. In 2017, I was kind of like, I was the intern on this Transformer paper, uh, and I thought, no, this is just research. You know, we just create new architectures, improve translation scores by three percent, and that's, that's what it is. Um, I didn't expect all that came of that, that architecture, uh, the Transformer and, um, the community's love for it and, and real, like, consolidation onto the Transformer as a platform for building AI. That I didn't expect. Um, With language modeling and the whole scaling project, I thought the world would catch on way faster to that piece. It started to become really obvious, but then it was two, three years before everyone woke up, and it sort of hit the world.

AI assessment note: “No, not, not at that moment. In 2017, I was kind of like”

Q Will we live in this world of unbundled, verticalized models, which are much more efficient and smaller, designed for specific use cases, or will there be much larger three to five models, which kind of rule it all?

A There will be both. There'll be both. Like, the one pattern I think we've seen emerge over the past Couple years. Is that people love prototyping with a generally smart model. They don't want to prototype with a specific model. They don't want to spend the time fine tuning a model to make it specifically good at the thing that they care about. What they want to do is just grab, you know, an expensive big model, prototype with that, prove that it can be done, and then distill that into an efficient focus model at the specific thing they care about. So that, that pattern has really emerged. So I, I think we'll Continuously exist in a world of multiple models, some focused and verticalized, others completely horizontal.

AI assessment note: “There will be both. There'll be both.”

Q Yeah. And so you were expecting it to pop off relatively quickly, I take it?

A No, not, not at that moment. In 2017, I was kind of like, I was the intern on this Transformer paper, uh, and I thought, no, this is just research. You know, we just create new architectures, improve translation scores by three percent, and that's, that's what it is. Um, I didn't expect all that came of that, that architecture, uh, the Transformer and, um, the community's love for it and, and real, like, consolidation onto the Transformer as a platform for building AI. That I didn't expect. Um, With language modeling and the whole scaling project, I thought the world would catch on way faster to that piece. It started to become really obvious, but then it was two, three years before everyone woke up, and it sort of hit the world.

AI assessment note: “No, not, not at that moment. In 2017”

Q the deep end bluntly, because I think it's a question that everyone's asking, which is like, everyone just says just throw more compute, and that is the single biggest rate limit that we have today. We just need more compute and performance will increase. Do you think that is true? There is a lot more room to run there, or it is other elements that are now holding back performance?

A I mean, it's definitely true that if you throw more compute at the model, if you make the model bigger, it'll get better. It's kind of like, it's the most trustworthy way to improve models. It's also the dumbest, right? Like, if all else fails, just make it bigger. Uh, and so for folks who have a lot of money, that's a really compelling strategy. It's super low risk. You know it's gonna get better. Just scale the model up, pay more money, pay, pay for more compute, and go. But yeah, I, I mean, I, I believe in it. I just think it's, Uh, extremely inefficient. There, there are much better ways. If you look at the past, let's say like a year and a half, so between, I guess by now it would be like between ChatGPT coming out or, or GPT-IV coming out, uh, and now. GPT-IV, if it's true what they say, and it's 1.7 trillion parameters, this big MOE, we have models that are better than that model that are like, thirteen billion parameters. Right? And so the scale of change, uh, like how quickly that became cheaper, is absurd. Like, it's, it's kind of surreal. Um, and so yes, you, you can achieve that quality of model just by scaling, but you probably shouldn't.

AI assessment note: “it's definitely true that if you throw more compute at the model... it'll get better”

Q Why is that so hard, and why do we not have any notion of that today?

A I think it's not that reasoning is hard. Uh, I think it's that, um, there's not a lot of training data that demonstrates reasoning out on the internet. Um, the internet is a lot of the output of a reasoning process. Like, you don't show your work when you're writing something on the web. You sort of present your conclusion, uh, or present your idea, which is the output of loads of thinking and, and experience and discussion. Um, So we just lack the training data. It's just not freely available. You have to build it yourself. And so that's what companies like Cohere and OpenAI and Anthropic, etc. That's what we're doing now is collecting data that demonstrates human reasoning.

AI assessment note: “there's not a lot of training data that demonstrates reasoning out on the internet.”

Q compression because they don't charge more per seat, but they have AI infused in all of their product, and so you can create, you know, anything with AI in their products, and obviously each query costs money. And so it's costing them more money and they're making the same revenue. Will we actually be able to make more revenue per user or will it just create a better customer experience?

A There's two different camps right now. Some people are pricing the exact same with AI features and using it to drive expansion in their business. And then the other folks like Microsoft, like Salesforce, like, uh, like Notion as well, um, they're charging for the AI features and getting a bigger business as a product. Both of those strategies are, are fine and super reasonable. For folks like Canva who are keeping the same price, um, I mean, I think it's a good bet. They want to grow their user base. They want to expand their user set. Just give them the most useful product possible. At the moment, don't worry about margins because the cost of AI is falling super, super quickly. Um, I think that's reasonable.

AI assessment note: “There's two different camps right now. Some people are pricing the exact same”

Q Does that continue in that same progressive advancement? And what I mean by that is, do we continue to see that same scaling advantages, or does it actually plateau at some point? As you said there, you know, we always hear about Moore's Law. At some point, It just becomes a better calculator for the iPhone.

A Yeah, I mean, I think it certainly requires exponential input. You know, you need to continuously be doubling your compute in order to sustain linear gains in, in intelligence. Um, but I, I think that probably goes on for a very, very, very long time. Um, it'll just keep getting smarter, but you run into, like, economic constraints, right? Uh, not a lot of people bought the original GPT-IV. Certainly not a lot of enterprises, because it was huge. It was massive. Super inefficient to serve. So costly. Not smart enough to justify that cost. And so I think there's a lot of pressure on making smaller, more efficient models. Smarter via data and algorithms, methods, rather than just scaling up due to market forces, just pressure on price.

AI assessment note: “I think that probably goes on for a very, very, very long time.”

Q There's this weird understanding that I have now from meeting so many incredible founders, and it's this incredibly high correlation between those that gamed in their early years and those that achieved success. I just, why do you think gaming is such a contributor to successful founders?

A Video games teach something to you. Um, you're much more Willing to grind. Right. To just do repetitive, difficult, painful things towards some broader goal. So that sort of resilience, um, I think is important. And then also the, the fact that you, you can respond. Like you get to try again. You get a second attempt. That optimism or that, that framing is really important. I think in a lot of cultures, you get one shot. You know, you have, you have a reputation, and if you fuck it, it's done. It's over for you. But maybe what gaming can give people is a sense of, you can fuck up, and you can try again, and you can get better. And the second time, you fuck up less than the first time, and the third time, you fuck up less than the second time. And so that notion of progress through failure, I think, is probably something very significant for founders.

AI assessment note: “you're much more Willing to grind. Right. To just do repetitive, difficult, painful things”

Q You mentioned the word agent there. Agents is one of the kind of hottest topics in, in venture land. Do you think it's justified, the hype around agents, agentic behavior, what it does to workflows?

A I mean, the hype is justified a hundred percent. Like, this is, that's the promise of, of AI. The promise of these models is that they would be able to carry out work by themselves. Um, That just dramatically transforms productivity. Once you have a model that can go off and, and do things independently over a very long time horizon. So no longer like I'm going to do this one thing for you immediately in return and I'm done. But like over the next six months, I'm going to be, you know, pumping deals into your, your top of funnel or something like that, right? Like pulling, pulling, doing outbound for you. It just completely transforms what an organization can do. Um, so the, the hype is justified. I think my critique would be, is that work gonna be most effectively done outside the model builders or within? Who's gonna be best positioned to actually build that product?

AI assessment note: “the hype is justified a hundred percent”

Q Uh, everyone talks about the death of Europe. You know, I had the wonderful Delian from Founders Fund on saying that Western Europe will be a third world state or kind of collection of countries soon, and negativity is quite real here, it feels. How do you feel now building incredible engineering research teams in London and Europe?

A England stands out from the rest of Europe. Um, There's a technology optimism that exists here, and a willingness to invest and make the changes necessary to support developing an ecosystem. In Europe proper, and by the way, my, my mom is British, my dad is Spanish, uh, and I have both citizenship, so I'm, I'm also very much European, spent my summers there, like, I, family is there. Um, unfortunately, The culture is just hostile towards tech. It's hostile. Like the, the solution to tech is regulation in the European mind. I think there's pressure to change though, and France is becoming much more ambitious and making a lot of noise on the European stage as well as the global stage about we need to be more progressive. It might take a decade though.

AI assessment note: “There's a technology optimism that exists here, and a willingness to invest”

Q the deep end bluntly, because I think it's a question that everyone's asking, which is like, everyone just says just throw more compute, and that is the single biggest rate limit that we have today. We just need more compute and performance will increase. Do you think that is true? There is a lot more room to run there, or it is other elements that are now holding back performance?

A I mean, it's definitely true that if you throw more compute at the model, if you make the model bigger, it'll get better. It's kind of like, it's the most trustworthy way to improve models. It's also the dumbest, right? Like, if all else fails, just make it bigger. Uh, and so for folks who have a lot of money, that's a really compelling strategy. It's super low risk. You know it's gonna get better. Just scale the model up, pay more money, pay, pay for more compute, and go. But yeah, I, I mean, I, I believe in it. I just think it's, Uh, extremely inefficient. There, there are much better ways. If you look at the past, let's say like a year and a half, so between, I guess by now it would be like between ChatGPT coming out or, or GPT-IV coming out, uh, and now. GPT-IV, if it's true what they say, and it's 1.7 trillion parameters, this big MOE, we have models that are better than that model that are like, thirteen billion parameters. Right? And so the scale of change, uh, like how quickly that became cheaper, is absurd. Like, it's, it's kind of surreal. Um, and so yes, you, you can achieve that quality of model just by scaling, but you probably shouldn't.

AI assessment note: “yes, you, you can achieve that quality of model just by scaling, but you probably shouldn't.”

Q Why is that so hard, and why do we not have any notion of that today?

A I think it's not that reasoning is hard. Uh, I think it's that, um, there's not a lot of training data that demonstrates reasoning out on the internet. Um, the internet is a lot of the output of a reasoning process. Like, you don't show your work when you're writing something on the web. You sort of present your conclusion, uh, or present your idea, which is the output of loads of thinking and, and experience and discussion. Um, So we just lack the training data. It's just not freely available. You have to build it yourself. And so that's what companies like Cohere and OpenAI and Anthropic, etc. That's what we're doing now is collecting data that demonstrates human reasoning.

AI assessment note: “there's not a lot of training data that demonstrates reasoning out on the internet”

Q What do they just get totally wrong about AI? I think the enterprise, um, education curve is still very early. What do they just not understand about it?

A There's a lot of fear around AI being wrong. There's like hallucination in, in these models, and everyone views that as some sort of like, you know, the technology is doomed. You know, sometimes it hallucinates. It doesn't reflect reality. The models definitely do hallucinate. The hallucination rates have been dropping dramatically, but they'll, they'll always have some chance of making stuff up or getting something wrong. But we exist in a world with humans, and humans hallucinate. Constantly. We get stuff wrong, we, you know, misremember things, and so we exist in a world that's robust to error, and so I, I think there's too much emphasis on that. We don't have hallucination benchmarks though, do we? We do, yeah.

AI assessment note: “There's a lot of fear around AI being wrong.”

Q There's this weird understanding that I have now from meeting so many incredible founders, and it's this incredibly high correlation between those that gamed in their early years and those that achieved success. I just, why do you think gaming is such a contributor to successful founders?

A Video games teach something to you. Um, you're much more Willing to grind. Right. To just do repetitive, difficult, painful things towards some broader goal. So that sort of resilience, um, I think is important. And then also the, the fact that you, you can respond. Like you get to try again. You get a second attempt. That optimism or that, that framing is really important. I think in a lot of cultures, you get one shot. You know, you have, you have a reputation, and if you fuck it, it's done. It's over for you. But maybe what gaming can give people is a sense of, you can fuck up, and you can try again, and you can get better. And the second time, you fuck up less than the first time, and the third time, you fuck up less than the second time. And so that notion of progress through failure, I think, is probably something very significant for founders.

AI assessment note: “Video games teach something to you. Um, you're much more Willing to grind.”

Q Will we live in this world of unbundled, verticalized models, which are much more efficient and smaller, designed for specific use cases, or will there be much larger three to five models, which kind of rule it all?

A There will be both. There'll be both. Like, the one pattern I think we've seen emerge over the past Couple years. Is that people love prototyping with a generally smart model. They don't want to prototype with a specific model. They don't want to spend the time fine tuning a model to make it specifically good at the thing that they care about. What they want to do is just grab, you know, an expensive big model, prototype with that, prove that it can be done, and then distill that into an efficient focus model at the specific thing they care about. So that, that pattern has really emerged. So I, I think we'll Continuously exist in a world of multiple models, some focused and verticalized, others completely horizontal.

AI assessment note: “There will be both. There'll be both.”

Q How do you think about competing against, like, OpenAI's incredible UGC play?

A Yeah, no, that's super difficult. And especially with enterprises, they never let you train on their data. And so we can't train on any of our customers' data. Super private. Their perspective is their data is their IP. There's too many secrets in their IP. And so they're just not willing to do it. And I, I'm super empathetic to that position. Um, and so for us, our focus is synthetic data. We push a lot on that as well as having like a human annotation force and scale as a partner for that. We have our own folks in house, but that's the burden that's placed on us because we're not a consumer company. We have to generate this data ourselves. The benefit is We're more focused. So we have less surface area to cover. So it's not the entire world showing up and asking us to do potentially anything. It's like enterprises with very clear patterns for the type of stuff they want to do. It's like they want to automate certain finance functions, or they want to automate certain, you know, HR functions. And so the scope is reduced dramatically, which lets us really focus in on, on those pieces.

AI assessment note: “for us, our focus is synthetic data. We push a lot on that”

Q You mentioned the word agent there. Agents is one of the kind of hottest topics in, in venture land. Do you think it's justified, the hype around agents, agentic behavior, what it does to workflows?

A I mean, the hype is justified a hundred percent. Like, this is, that's the promise of, of AI. The promise of these models is that they would be able to carry out work by themselves. Um, That just dramatically transforms productivity. Once you have a model that can go off and, and do things independently over a very long time horizon. So no longer like I'm going to do this one thing for you immediately in return and I'm done. But like over the next six months, I'm going to be, you know, pumping deals into your, your top of funnel or something like that, right? Like pulling, pulling, doing outbound for you. It just completely transforms what an organization can do. Um, so the, the hype is justified. I think my critique would be, is that work gonna be most effectively done outside the model builders or within? Who's gonna be best positioned to actually build that product?

AI assessment note: “I mean, the hype is justified a hundred percent.”

Q Uh, everyone talks about the death of Europe. You know, I had the wonderful Delian from Founders Fund on saying that Western Europe will be a third world state or kind of collection of countries soon, and negativity is quite real here, it feels. How do you feel now building incredible engineering research teams in London and Europe?

A England stands out from the rest of Europe. Um, There's a technology optimism that exists here, and a willingness to invest and make the changes necessary to support developing an ecosystem. In Europe proper, and by the way, my, my mom is British, my dad is Spanish, uh, and I have both citizenship, so I'm, I'm also very much European, spent my summers there, like, I, family is there. Um, unfortunately, The culture is just hostile towards tech. It's hostile. Like the, the solution to tech is regulation in the European mind. I think there's pressure to change though, and France is becoming much more ambitious and making a lot of noise on the European stage as well as the global stage about we need to be more progressive. It might take a decade though.

AI assessment note: “England stands out from the rest of Europe. Um, There's a technology optimism that exists here”

Q What do they just get totally wrong about AI? I think the enterprise, um, education curve is still very early. What do they just not understand about it?

A There's a lot of fear around AI being wrong. There's like hallucination in, in these models, and everyone views that as some sort of like, you know, the technology is doomed. You know, sometimes it hallucinates. It doesn't reflect reality. The models definitely do hallucinate. The hallucination rates have been dropping dramatically, but they'll, they'll always have some chance of making stuff up or getting something wrong. But we exist in a world with humans, and humans hallucinate. Constantly. We get stuff wrong, we, you know, misremember things, and so we exist in a world that's robust to error, and so I, I think there's too much emphasis on that. We don't have hallucination benchmarks though, do we? We do, yeah.

AI assessment note: “There's a lot of fear around AI being wrong.”

Q How do you think about competing against, like, OpenAI's incredible UGC play?

A Yeah, no, that's super difficult. And especially with enterprises, they never let you train on their data. And so we can't train on any of our customers' data. Super private. Their perspective is their data is their IP. There's too many secrets in their IP. And so they're just not willing to do it. And I, I'm super empathetic to that position. Um, and so for us, our focus is synthetic data. We push a lot on that as well as having like a human annotation force and scale as a partner for that. We have our own folks in house, but that's the burden that's placed on us because we're not a consumer company. We have to generate this data ourselves. The benefit is We're more focused. So we have less surface area to cover. So it's not the entire world showing up and asking us to do potentially anything. It's like enterprises with very clear patterns for the type of stuff they want to do. It's like they want to automate certain finance functions, or they want to automate certain, you know, HR functions. And so the scope is reduced dramatically, which lets us really focus in on, on those pieces.

AI assessment note: “We're more focused. So we have less surface area to cover.”

Q was brought up on the idea that actually we always overestimate things in the short term and underestimate them in the long term. To what extent do you think that's the case here? Or actually, voice is coming and coming pretty quickly. GPT-V is coming and coming, you know, whether that's in three to six months, still coming pretty quickly. To what extent are we actually underestimating the short term?

A I think it's, it's, okay, there's like two, two things happening. One, it's, it's getting harder It's getting harder to deliver gains in the models. It is getting more difficult, more arduous, more costly, because there was a time where the models were dumb enough that I could pull. I say dumb enough, uh, but the models were- Not sophisticated. Yeah, sufficiently, uh, unintelligent that I could pull anyone off the street. Any human was more intelligent than the model and had something to teach it, right? I could just grab someone, say, talk to this model, find errors, and they will. And improve it. Eventually, uh, the models, like, it was just kind of hard to get people to the average person to find knowledge gaps or that type of thing, and so you had to start going to domain experts. Um, and initially, like, cheap kind of junior ones, like students of computer science could teach the model something, students of biology could teach the model something, and then the models started getting really good and kind of matching that level of knowledge. And you're, you're just going into more specific and more scarce pools of talent to get them to teach the model their knowledge. Um, and so it gets more high friction, more expensive to, to teach the model the incremental new knowledge.

AI assessment note: “It's getting harder to deliver gains in the models. It is getting more difficult”

Q was brought up on the idea that actually we always overestimate things in the short term and underestimate them in the long term. To what extent do you think that's the case here? Or actually, voice is coming and coming pretty quickly. GPT-V is coming and coming, you know, whether that's in three to six months, still coming pretty quickly. To what extent are we actually underestimating the short term?

A I think it's, it's, okay, there's like two, two things happening. One, it's, it's getting harder It's getting harder to deliver gains in the models. It is getting more difficult, more arduous, more costly, because there was a time where the models were dumb enough that I could pull. I say dumb enough, uh, but the models were- Not sophisticated. Yeah, sufficiently, uh, unintelligent that I could pull anyone off the street. Any human was more intelligent than the model and had something to teach it, right? I could just grab someone, say, talk to this model, find errors, and they will. And improve it. Eventually, uh, the models, like, it was just kind of hard to get people to the average person to find knowledge gaps or that type of thing, and so you had to start going to domain experts. Um, and initially, like, cheap kind of junior ones, like students of computer science could teach the model something, students of biology could teach the model something, and then the models started getting really good and kind of matching that level of knowledge. And you're, you're just going into more specific and more scarce pools of talent to get them to teach the model their knowledge. Um, and so it gets more high friction, more expensive to, to teach the model the incremental new knowledge.

AI assessment note: “It is getting more difficult, more arduous, more costly”

Q Can we just, what are data innovations, and what are model and method innovations?

A Yeah, so, uh, Pretty much all of the major gains that we've seen in the, uh, open source space have come from data improvements. Uh, so models getting much better by taking higher quality data, uh, from the internet, better scraping algorithms, parsing those web pages, pulling out the right parts, up weighting specific parts of the internet, because there's lots of, like, repetition and junk, right? And so pulling out the most valuable Um, knowledge-rich parts of the internet and emphasizing them to the model. Synthetic data and the ability to create new data that is super scalable, so you can get many, many billions of words or, you know, hundreds of millions of pages of this stuff, but it's no humans involved, just written by models. Those innovations, the ability to increase the quality of data, have led to, I think, most of the gains. That we're, we're seeing right now.

AI assessment note: “all of the major gains that we've seen... have come from data improvements.”

Q Can we just, what are data innovations, and what are model and method innovations?

A Yeah, so, uh, Pretty much all of the major gains that we've seen in the, uh, open source space have come from data improvements. Uh, so models getting much better by taking higher quality data, uh, from the internet, better scraping algorithms, parsing those web pages, pulling out the right parts, up weighting specific parts of the internet, because there's lots of, like, repetition and junk, right? And so pulling out the most valuable Um, knowledge-rich parts of the internet and emphasizing them to the model. Synthetic data and the ability to create new data that is super scalable, so you can get many, many billions of words or, you know, hundreds of millions of pages of this stuff, but it's no humans involved, just written by models. Those innovations, the ability to increase the quality of data, have led to, I think, most of the gains. That we're, we're seeing right now.

AI assessment note: “Pretty much all of the major gains that we've seen in the, uh, open source space”

Q Do you think chat is the best interface for consumers?

A For some stuff, I think for other stuff, GUI, like, you know, like a user interface, the traditional visual one is quite good. Um, I think it really depends. Chat as an interface onto everything, I don't think makes sense. I don't want to have to type out explicitly my instructions to get stuff done. Like sometimes I just want to click some buttons and go through a GUI and get the job done. So yeah, I don't think like, you know, GUIs are dead and that we should replace everything with a text box. But I do think it provides this really compelling interface. Certainly voice does. Like voice, voice is Magical. Certainly, it was magical the first time I saw a model write text back to me as compellingly as a human. That happened, like, in 2017, shortly after we submitted the paper. We started training language models on Wikipedia, um, and we sampled from those models, and it could write Wikipedia pages as convincing as a human page. And so that, that was a very Magical moment that computers kind of woke up and started speaking back to us. Um, and then the next time was dialogue as an interface, so not just I submit an instruction, the model returns a response, but having a conversation over chat with the model.

AI assessment note: “Chat as an interface onto everything, I don't think makes sense.”

Q Do you think chat is the best interface for consumers?

A For some stuff, I think for other stuff, GUI, like, you know, like a user interface, the traditional visual one is quite good. Um, I think it really depends. Chat as an interface onto everything, I don't think makes sense. I don't want to have to type out explicitly my instructions to get stuff done. Like sometimes I just want to click some buttons and go through a GUI and get the job done. So yeah, I don't think like, you know, GUIs are dead and that we should replace everything with a text box. But I do think it provides this really compelling interface. Certainly voice does. Like voice, voice is Magical. Certainly, it was magical the first time I saw a model write text back to me as compellingly as a human. That happened, like, in 2017, shortly after we submitted the paper. We started training language models on Wikipedia, um, and we sampled from those models, and it could write Wikipedia pages as convincing as a human page. And so that, that was a very Magical moment that computers kind of woke up and started speaking back to us. Um, and then the next time was dialogue as an interface, so not just I submit an instruction, the model returns a response, but having a conversation over chat with the model.

AI assessment note: “Chat as an interface onto everything, I don't think makes sense.”