The Exchanges

Q go on Twitter, and there is this transience of dominance between the different providers. You know, OpenAI do something and it's like, wow, that's incredible. And then Claude do something and it's like, wow, that's incredible. Llama, what? And every week it seems like this one's the winner and the rest are losing. And there's just such transience and speed. I almost don't know where to go. Is that understandable?

A It doesn't help that Social media likes buzz for normal people sitting back. Everyone's just gonna keep using chat GPT. Cause that's what they're using, right? They might gradually switch to cloud. Like the enthusiast community is very different than when I talk to the outside world about this stuff. And I think that on the grand sweep of things, what really matters is when these models top out, um, and how long that takes. And I think worrying about who's in the lead at one moment is probably less of an issue. Then the big labs are all gonna keep building. There's no tricks in llama that they really told us that were unusual or Indicated some sort of secret breakthrough. We still don't know if there's secret sauce in some of the other labs that are very different. Like it's a very early days in some ways. So I think trying to get, you know, if you're enthusiastic like me about this technology, great. Follow along and keep, keep track of the, you know, MLA ratings. But otherwise I do think there's a little bit of like, You know, unnecessary to get every detail at this stage.

AI assessment note: “worrying about who's in the lead at one moment is probably less of an issue”

Q The final one before we do a quick fire, uh, a friend of mine, who's also a quite a well-known venture capitalist, Jeff Lewis said that when it comes to democracy in the future, we will vote for algorithms, not for people. To what extent do you think AI pervades into electoral systems, electoral voting, the political fabric of our society?

A When something feels like a dystopia to most people, it probably is something that's not going to happen very quickly. Um, you know, human systems are complicated. I, I just keep seeing this technological view, which is like, you know, in a rational world, the machines will rule us all. It's just, people don't want that, right? So like, you know, we already have algorithms ruling lots of what we do. You know, your FICO score determines a huge amount of, of, you know, things that happen in your life. Um, and that's an algorithm. Like we have these kinds of systems in place, but the idea of an overall all seeing kind of approach, it's, it's hard. Like now on the other hand, we do find that AI is hyper persuasive already, right? In a controlled experiment where you do, where you're apps to be, you talk to a normal person versus the AI, you're 81.7% more likely to change your views, the AI's view than to a human's view. That is going to change marketing in very big ways, which is going to change politics. Right. Deep fakes are going to our big deal already. Although it's been funny how little big, a big deal they are, because it just turns out all you need to do is show a video of politician X talking and say, I can't believe he said he's going to eat babies in minute three. And everybody shares it online who should know better. And without actually watching the video at all, like …

AI assessment note: “That is going to change marketing in very big ways, which is going to change politics.”

Q Can I ask you on the flip side, we have the companies themselves. What are companies getting wrong about AI that they should know more about?

A I mean, I speak to organizations all the time. I mean, first of all, just from a perspective, almost nobody uses these systems. I mean, they all tried chat TPT, right? Every, when I asked my hand, everybody's tried chat TPT, almost always the three, five version of before about Five to 10% of people in any room, whether, by the way, Silicon Valley, actual people, right, who aren't at a lab, whether that's at a large bank, whether that's at a conference of innovation professionals, maybe five to 10% have used those models, and maybe two or three percent have used 10 hours, which has been my, you know, sort of guideline, you know, minimum number. And I think, again, there's no onboarding. You're faced with a chatbot, and when people are faced with the tyranny of the blank page, They panic. What do you talk to the system about? Right? And like, there's no information. There's no instructions. And so people aren't really using it. So the issue is that partially it's that they need to adopt because when people start using it, they find uses, right? So a new study just came out of Denmark of people who are using chat GPT and, you know, in knowledge intensive work environments. And, you know, they're estimating that in, you know, over 30% of their tasks, they're saving 50% of their time. So once people use it, they find productive uses. So then the question becomes, How are you harnes…

AI assessment note: “almost nobody uses these systems. I mean, they all tried chat TPT, right?”

Q How much of that chasm do you think will be closed by the closed source providers with their next releases?

A I think we don't know a lot, and even the people train the models don't know a lot. I mean, part of the weird bit here, right, is the people train the models are all computer scientists, basically. I mean, doing computer science, And they don't have a huge idea of the implications of the systems. When open AI released GPT, 3.5, they didn't expect to destroy higher education, you know, education, and then we'd have to rebuild it because everyone's cheating all of a sudden, right? I mean, they were already cheating, but now they're just cheating really well, but we weren't expecting like a large scale revision of like how the world works, right? And so I think we don't know. I think the model, everything I'm hearing from everybody is that the next generation of models is going to be smarter, right? The exponential continues. Whether or not that translates to the real world implications, a different kind of concern.

AI assessment note: “everything I'm hearing from everybody is that the next generation of models is going to be smarter”

Q Is that really an order of magnitude improvement if we compare that post-classroom? You could give me incredible high-quality videos of you Talking, lecturing, giving examples that you give to your students now, very easy to do, versus that AI tutor. Is it, is it 20% better? Sure, maybe it's personalized, but is it really an order of magnitude better?

A Education is a complex system. So I think order of magnitude is a very weird thing to talk about, because every student has their own talents, abilities, interests, and gaps. The early work on, in one-on-one tutoring, we don't talk about order of magnitude improvement, because that doesn't really work in the education world. It's very hard to say what an order of magnitude is. But we could talk about grades a lot and the classic study that is probably would not be replicable, but it sets up our model is that one-on-one tutoring, um, according, it creates a two sigma increase in, uh, in classroom outcomes. That's two standard deviations, which is, you know, a fairly huge improvement. You go from the 50th percentile to the 97th percentile in class. We have no idea if that's gonna hold up with, you know, AI tutoring, but if we could do that, that is as amazing an improvement as you could possibly ask for. I mean, a 10% improvement is amazing. I, I kind of feel like aiming for order of magnitude education, if we can get improvement in a system, we're in great shape.

AI assessment note: “order of magnitude is a very weird thing to talk about”

Q What is a good model for a radical regime, then? Because I've been brought up, quite rightly, as you mentioned there, in the incremental innovation kind of economy, where it's, like, Test, iterate, find product, market fit. Someone pays for it. Good. Well done. So what is the right model in this new age of kind of radical innovation shift?

A VCs have funded this model, right? And it's like deep tech, medical, like things where you're making larger bets in the future, where they're, you know, where there's payoff is where, where, when it's revealed to the world is either going to succeed or not. Right. And where you're making a better technology itself, that's where VC got its start. And it's sort of, Became, you know, perverted a little bit to this, like, how do I get, you know, make money fast machine? I mean, not that fast, right? It's still years till exit, but there's the idea of like, you know, with, you know, it, it, it, it's all about prorata rights and the idea of like, I make a lot of small bets initially, and then I can double down on the people doing well and not do double down on others. And it's about finding the diamond in the rough. Like all of that stuff is like a great model for funding incremental innovation. If the market's changing, like, and we're used to market changing slowly enough that like, Like, that's not a problem. I think it's an issue here. I think you need to be imaginative. I think you need to be subject specific. I need, think you need to assume model. I mean, it is very strange from one hand for all of these people in Silicon Valley to be like, yeah, you know, AGI is coming. And then the applications they're building are like these very narrow, like, hey, I slapped something on to…

AI assessment note: “deep tech, medical, like things where you're making larger bets in the future”

Q For those that don't understand, why will none of them survive in an AGI world, Ethan?

A So, the common definition of AGI is a machine that's smarter than humans at every task. So, like, the machine will decide what to do. You're not, like, who cares about your stupid product, right? Like, you've been making this for humans to and getting a product market fit, and then, but the humans will say, you know, optimize my trading strategy, or the AI will just decide to optimize your trading strategy. I mean, no one knows what AGI looks like, so I'm not going to try and paint a science fiction future, But I will say there's a huge contradiction between a market race and saying AGI soon. And like, we're funding a bunch of companies that are helping, like, you know, already, I don't know if you've played with them, like, not that we're in anywhere near AGI with this, but you know, you could tell Claude come up with 30 ideas for a product to serve, you know, market X, then rate them all on quality and feasibility level. Then create, this is one prompt, by the way, then create a, uh, a playable prototype of the interface for the application. Then interview me as a user about how to change it and adapt it as we go. And it does it. Like, I get a little playable interface for a game, and I can then edit the game and, you know, say, like, oh, I wish it was more, you know, it's, it's, it's just not fun enough in some way. And it's like, okay, great. I'll make it more fun for you. …

AI assessment note: “the machine will decide what to do. You're not, like, who cares about your stupid product”

Q compute will solve all problems and many have always believed that. That performance will be answered by compute and just more brute compute. I have other people on the show. You're Alex Wang's at scales who say that data is the core bottleneck. When we think about compute data or algorithms, what do we think is the core bottleneck to performance now and in the next 12 to 24 months?

A I'll try and answer that, but I want to take the contrarian view first that I always want to indicate first, which is for most people, they just don't care, right? Like, Let's say that LLM's top out, and it turns out we have to switch to, you know, Mamba or some other, like, you know, uh, other, like, who cares? Nobody cares. They're using these systems. We don't know what, like, mixture of experts. Like, there's a lot of, like, in the weeds that you get when you're watching this, like, uh, like a sports game of, like, who's winning and what situation that, like, the top line capabilities matter, and there's a lot of room left there. Like, to me, the thing that gets left on these computer science discussions are often the system, the human systems that these things have to interact with. The organizational systems they have to interact with, and that's where we need to see kind of more growth, right? That being said, we don't know what the bottleneck is, right? There's, there's this idea in the history of science called the reverse salient, which is that technology sort of moves forward, but there's always something that's kind of lagging, and all the effort goes into fixing the lag. So in the early days of electricity, we had generators, but transmission was a problem. So it was a huge amount of work to make transition better, and our current electrical sort of New economy. It…

AI assessment note: “we don't know what the bottleneck is, right?”

Q You said there about, hey, you know, people work on small kind of minute things on top of Lama say, and then it's like, well, you need to be opinionated about who you're going after and who you're not going after. You need to be more targeted. Is that not kind of one in the same, which is like the verticalization of approach and the targeted approach being the core?

A Well, I think it's not about verticalization as much as opinionated, right? I think you need to have a strong opinion of what the future looks like and where the gaps are going to remain. This is a jagged technology, trying to work on the exact technique, like figure out where you think there's going to be jaggedness and that can be organizational jaggedness, interface jaggedness. But I mean, you're also basically the real problem right now is every startup in the world is betting against, um, AGI. Which I find really funny because all the funders are like, yeah, AGI is coming in the next five years. If it is, why are you funding these startup companies? Like none of them are surviving in AGI world.

AI assessment note: “Well, I think it's not about verticalization as much as opinionated, right?”

Q I also think that doesn't include a lot of different elements that you mentioned that extrinsic motivation being a big part of it. I think a big part of like having a tutor means you actually have a bond with them. You want to impress them. You want them to feel proud of you. Does that extend to an AI tutor where you don't have that humor?

A Maybe. I mean, we actually, it's not clear that that is the key to tutoring is the bond with the human being. It, it, it seems to be that across a wide variety of tutoring approaches, That forcing people to confront what they don't know turns out to be a lot of the value of tutoring. So tutoring is also often reflective back. So it's like, how do you, so when we built a tutor, um, a tutor chat bot, right? What that tutor chat bot, like the way we test, by the way, education, technology chat bots, our rule of thumb is that if it asks you, if you understand a topic or you're ready to move on, it's a bad tutor because humans don't know when they're ready to move on or not. What the AI should be doing is asking you questions, probing what you know, And making you expand on what you don't understand and helping you fill those gaps. So it's not the one-on-one bond. There is a, there is, there are methods to teaching that we actually know make a difference. Self-reflection makes a difference, right? Repeated practice makes a difference. Low stakes testing makes a difference. Like there, this, and this is kind of like to zoom back out to what we were talking about before. Subject matter expertise is going to be absolutely critical in making AI work. It's a system that experts I can look at a prompt in entrepreneurship and education and instantly tell you about whether that's going to w…

AI assessment note: “it's not clear that that is the key to tutoring is the bond”

Q For those that don't understand, why will none of them survive in an AGI world, Ethan?

A So, the common definition of AGI is a machine that's smarter than humans at every task. So, like, the machine will decide what to do. You're not, like, who cares about your stupid product, right? Like, you've been making this for humans to and getting a product market fit, and then, but the humans will say, you know, optimize my trading strategy, or the AI will just decide to optimize your trading strategy. I mean, no one knows what AGI looks like, so I'm not going to try and paint a science fiction future, But I will say there's a huge contradiction between a market race and saying AGI soon. And like, we're funding a bunch of companies that are helping, like, you know, already, I don't know if you've played with them, like, not that we're in anywhere near AGI with this, but you know, you could tell Claude come up with 30 ideas for a product to serve, you know, market X, then rate them all on quality and feasibility level. Then create, this is one prompt, by the way, then create a, uh, a playable prototype of the interface for the application. Then interview me as a user about how to change it and adapt it as we go. And it does it. Like, I get a little playable interface for a game, and I can then edit the game and, you know, say, like, oh, I wish it was more, you know, it's, it's, it's just not fun enough in some way. And it's like, okay, great. I'll make it more fun for you. …

AI assessment note: “common definition of AGI is a machine that's smarter than humans at every task”

Q What is a good model for a radical regime, then? Because I've been brought up, quite rightly, as you mentioned there, in the incremental innovation kind of economy, where it's, like, Test, iterate, find product, market fit. Someone pays for it. Good. Well done. So what is the right model in this new age of kind of radical innovation shift?

A VCs have funded this model, right? And it's like deep tech, medical, like things where you're making larger bets in the future, where they're, you know, where there's payoff is where, where, when it's revealed to the world is either going to succeed or not. Right. And where you're making a better technology itself, that's where VC got its start. And it's sort of, Became, you know, perverted a little bit to this, like, how do I get, you know, make money fast machine? I mean, not that fast, right? It's still years till exit, but there's the idea of like, you know, with, you know, it, it, it, it's all about prorata rights and the idea of like, I make a lot of small bets initially, and then I can double down on the people doing well and not do double down on others. And it's about finding the diamond in the rough. Like all of that stuff is like a great model for funding incremental innovation. If the market's changing, like, and we're used to market changing slowly enough that like, Like, that's not a problem. I think it's an issue here. I think you need to be imaginative. I think you need to be subject specific. I need, think you need to assume model. I mean, it is very strange from one hand for all of these people in Silicon Valley to be like, yeah, you know, AGI is coming. And then the applications they're building are like these very narrow, like, hey, I slapped something on to…

AI assessment note: “I think you need to be imaginative. I think you need to be subject specific.”