The Exchanges

Q What are we not talking or seeing in inference that we need to spend or think more about?

A You know, I think the thing that most people miss, uh, although the, um, Deep Seek R one launch, uh, you know, a few weeks back kind of clued everybody into it is that we just have an incredible track record over the past handful of years. And it's many years now of like just repeated year over year, like mind boggling progress and optimizing the performance of models so that performance of inference is just better and better and better. So like over time, The models have gotten bigger and the API calls have gotten cheaper. Uh, and like a little bit of that is because you get, you know, maybe like a two X benefit price performance from hardware every generation. Like if you're lucky, um, but you get a much bigger improvement, uh, price performance wise from. All of the things that you're doing in the software stack, you know, again, there, there's just a ton of work happening there. Um, You know, the DeepSea car one stuff, uh, which, which was good work, uh, is, you know, the way you should think about it is it's like a point on a line of, uh, price performance improvement that maybe was invisible to everyone else, but like not invisible to the people who are like, you know, neck deep and optimizing these systems. And it's not the last point. Like, you know, it, it just marches on.

AI assessment note: “the thing that most people miss, uh, although the, um, Deep Seek R one launch”

Q and I asked him this question, and I said, if we think about compute, uh, or kind of hardware, and then we think about models, and we think about apps, where does the value lie? And naturally, he said compute. Um, but when you think about that kind of three-pronged tier of value, and you said that models aren't products, if they're not products, Does that mean they're not valuable?

A No, no, no. They're super valuable, but they're only valuable to the extent that you can connect them to things that users need via product. So like in the limit, I think product is the most important thing. Now there, if you build good models and you build good infrastructure around models and you have good, efficient compute, and like you have all of these other things, you're going to get lots of ability to monetize all of those things because as As people build those products, like they will need to consume your platform and your infrastructure and like all of that's good. Uh, but most of the value has to be in the products. Like, you know, we don't build infrastructure just for the sake of infrastructure. Uh, like we build infrastructure so people can make product.

AI assessment note: “No, no, no. They're super valuable, but they're only valuable to the extent”

Q What are we not talking or seeing in inference that we need to spend or think more about?

A You know, I think the thing that most people miss, uh, although the, um, Deep Seek R one launch, uh, you know, a few weeks back kind of clued everybody into it is that we just have an incredible track record over the past handful of years. And it's many years now of like just repeated year over year, like mind boggling progress and optimizing the performance of models so that performance of inference is just better and better and better. So like over time, The models have gotten bigger and the API calls have gotten cheaper. Uh, and like a little bit of that is because you get, you know, maybe like a two X benefit price performance from hardware every generation. Like if you're lucky, um, but you get a much bigger improvement, uh, price performance wise from. All of the things that you're doing in the software stack, you know, again, there, there's just a ton of work happening there. Um, You know, the DeepSea car one stuff, uh, which, which was good work, uh, is, you know, the way you should think about it is it's like a point on a line of, uh, price performance improvement that maybe was invisible to everyone else, but like not invisible to the people who are like, you know, neck deep and optimizing these systems. And it's not the last point. Like, you know, it, it just marches on.

AI assessment note: “I think the thing that most people miss... repeated year over year... progress and optimizing”

Q and I asked him this question, and I said, if we think about compute, uh, or kind of hardware, and then we think about models, and we think about apps, where does the value lie? And naturally, he said compute. Um, but when you think about that kind of three-pronged tier of value, and you said that models aren't products, if they're not products, Does that mean they're not valuable?

A No, no, no. They're super valuable, but they're only valuable to the extent that you can connect them to things that users need via product. So like in the limit, I think product is the most important thing. Now there, if you build good models and you build good infrastructure around models and you have good, efficient compute, and like you have all of these other things, you're going to get lots of ability to monetize all of those things because as As people build those products, like they will need to consume your platform and your infrastructure and like all of that's good. Uh, but most of the value has to be in the products. Like, you know, we don't build infrastructure just for the sake of infrastructure. Uh, like we build infrastructure so people can make product.

AI assessment note: “No, no, no. They're super valuable, but they're only valuable to the extent that”

Q When you look forward to the next three to five years, Three to 10 years. How do you think about the pervasiveness of open versus closed and which will be more dominant than the other?

A I think there's gonna be lots of both. Um, And I, I think, you know, part of it is like, let's just like, forget about AI, which is sort of the controversial thing at the moment. It's like a thing where, you know, in industry structure hasn't settled yet. And like, we don't know exactly what it's going to be, but you just sort of pick your previous, uh, things like search, for instance, like there's a whole bunch of open source, uh, search engine projects out there. Um, and people who. Want to do search to like have a search feature in their application or who want to go build a search engine themselves have lots of options. Like they can go grab something open source as a starting point. They can stand up a product like they can go, you know, They can go load their data into something like Azure cognitive search, uh, which is a search as a service platform. Like Google has one. Amazon has one. Like they're, they're readily available. And then you still have search engines like, uh, being in Google, um, that are out there. And so like, they all exist. Um, all of the economics, uh, in search go to like somebody who's stood up a gigantic infrastructure and who's sort of running like a whole. Search business, uh, like with its own feedback loop. Um, yeah. And so I think, you know, we're probably gonna have similar sorts of things happening here. Um, for the infrastructure layer, y…

AI assessment note: “I think there's gonna be lots of both.”

Q When you look forward to the next three to five years, Three to 10 years. How do you think about the pervasiveness of open versus closed and which will be more dominant than the other?

A I think there's gonna be lots of both. Um, And I, I think, you know, part of it is like, let's just like, forget about AI, which is sort of the controversial thing at the moment. It's like a thing where, you know, in industry structure hasn't settled yet. And like, we don't know exactly what it's going to be, but you just sort of pick your previous, uh, things like search, for instance, like there's a whole bunch of open source, uh, search engine projects out there. Um, and people who. Want to do search to like have a search feature in their application or who want to go build a search engine themselves have lots of options. Like they can go grab something open source as a starting point. They can stand up a product like they can go, you know, They can go load their data into something like Azure cognitive search, uh, which is a search as a service platform. Like Google has one. Amazon has one. Like they're, they're readily available. And then you still have search engines like, uh, being in Google, um, that are out there. And so like, they all exist. Um, all of the economics, uh, in search go to like somebody who's stood up a gigantic infrastructure and who's sort of running like a whole. Search business, uh, like with its own feedback loop. Um, yeah. And so I think, you know, we're probably gonna have similar sorts of things happening here. Um, for the infrastructure layer, y…

AI assessment note: “I think there's gonna be lots of both.”

Q In those moments of transition where there is this confusion, what have you learned is the right action to do? Is it to be active, to iterate and learn, but you'll make so many mistakes that you regret? Or should you sit on your hands and watch others make those mistakes?

A Oh god, no. Like, you definitely shouldn't do the latter. Um, so, like, this is the best time to be alive if you have an entrepreneurial spirit. Um, and, like, the thing, the thing I think That you have to do in these moments is not forget. The things that you've learned from the past moments about what works. And it's not like, okay, well, like it's do this specific thing, but it's like how you go about doing, uh, you know, that exploration that you just described, which is. You know, product matters. Uh, yeah, I, I've been saying this for the past couple of years that models aren't products, uh, because everybody like was just so fascinated by the infrastructure itself. And, you know, like, oh, we, And, and like, this is also a characteristic of the beginning of these cycles is you have technical people who get just swept up in the. Technical bits. And they kind of forget that the only thing that really matters is making good product. And like, that's where we're at right now. Like you have to make good product and, um, you know, you have to have ideas and have conviction, and then you have to go get stuff done really fast so that you can see whether you're full of crap or not about the conviction that you have. And you're not. You have very few patterns at the beginning of a cycle to go snap to. Like, you're not looking at someone else's success and saying, okay, well, like,…

AI assessment note: “Oh god, no. Like, you definitely shouldn't do the latter.”

Q When we think about, like, the three core elements that, that make up, kind of, efficiency in this respect, it's kind of data computing algorithms. When we, like, drill into data, what are your biggest observations on data efficiency, the importance of quality of data versus quantity, synthetic versus human? How do you think about that today?

A Yeah, I mean, like, The, the mix of synthetic data is going up high quality data is becoming, uh, much more useful, uh, in, you know, especially in the post training parts of the. Um, model production pipeline, then low quality data. So, like, I, I think we're clearly at the point now where, you know, if you have. You know, if you have the right infrastructure and you have super high quality data and super high quality expert human feedback, you can amplify that into like the, the right set of tokens for, uh, training bigger and bigger models. And like that stuff is way more value than just, you know, sort of the undifferentiated tokens that are, you know, like floating around on, on the web.

AI assessment note: “mix of synthetic data is going up high quality data is becoming much more useful”

Q You mentioned earlier the centrality of products. Um, taking that into account with the current conversation here, How do we think about whether chat is the right UI for the next kind of paradigm of this product realm? You know, OpenAI and ChatGPT has made it the default. To what extent do you think it is the right default and how we will see that change?

A Like, I think it's a You know, reasonable step in the right direction. Like the thing I've been saying for a few years now is, uh, like, I think one of the most interesting things happening with AI is we we've had one paradigm for using computing devices for effectively 200 years since, uh, Ada Lovelace wrote the first program. So if you want a computing device to go do something for you, you had to be a programmer yourself, which is a pretty, you know, High barrier to entry for a lot of people, or you have to rely on the fact that a programmer has anticipated some need that you might have and like packaged up, uh, like a piece of software into an application that, that you were able to run. And those are the two ways you can get a computing device to do something for you, uh, until now. And so like the, the thing that changes with AI is, It can understand a thing that you want your computing device to go do for you. Uh, and it can figure out a way to go make that thing happen and you don't have to be a programmer. And, you know, it's, it's kind of a profound change because like, it basically means, and like, I don't think this is next year, but it's probably not going to be 10 years. Um, this whole notion that the, That, that you have teams of people who are, who, whose job is to go anticipate a bunch of very granular user needs in some narrow space, and then they're gonna go …

AI assessment note: “I think it's a You know, reasonable step in the right direction.”

Q When we think about, like, the three core elements that, that make up, kind of, efficiency in this respect, it's kind of data computing algorithms. When we, like, drill into data, what are your biggest observations on data efficiency, the importance of quality of data versus quantity, synthetic versus human? How do you think about that today?

A Yeah, I mean, like, The, the mix of synthetic data is going up high quality data is becoming, uh, much more useful, uh, in, you know, especially in the post training parts of the. Um, model production pipeline, then low quality data. So, like, I, I think we're clearly at the point now where, you know, if you have. You know, if you have the right infrastructure and you have super high quality data and super high quality expert human feedback, you can amplify that into like the, the right set of tokens for, uh, training bigger and bigger models. And like that stuff is way more value than just, you know, sort of the undifferentiated tokens that are, you know, like floating around on, on the web.

AI assessment note: “mix of synthetic data is going up high quality data is becoming, uh, much more useful”

Q What's the crazy AI prediction that most people would call science fiction that you believe to be true?

A It is already the case that I, I think the frontier models are probably better, uh, like, uh, health diagnosticians than Your average GP is it's a. Good thing to sort of realize and act on, uh, as quickly as possible because we have a whole world of people who have inadequate access to high quality health care, uh, including my own family in rural central Virginia, where, you know, it's just not, not good. Um, and so. Yeah, they're, they're just sort of a bunch of these things, uh, like this where. Yeah, the models are already really good and you've got. Um, you, you basically need the whole world to wake up to the fact that they're good, so that we can go deploy this stuff, and like, Deploy it because the thing that we really care about is the good of the public, uh, not trying to, you know, sustain some status quo.

AI assessment note: “frontier models are probably better, uh, like, uh, health diagnosticians than Your average GP”

Q In those moments of transition where there is this confusion, what have you learned is the right action to do? Is it to be active, to iterate and learn, but you'll make so many mistakes that you regret? Or should you sit on your hands and watch others make those mistakes?

A Oh god, no. Like, you definitely shouldn't do the latter. Um, so, like, this is the best time to be alive if you have an entrepreneurial spirit. Um, and, like, the thing, the thing I think That you have to do in these moments is not forget. The things that you've learned from the past moments about what works. And it's not like, okay, well, like it's do this specific thing, but it's like how you go about doing, uh, you know, that exploration that you just described, which is. You know, product matters. Uh, yeah, I, I've been saying this for the past couple of years that models aren't products, uh, because everybody like was just so fascinated by the infrastructure itself. And, you know, like, oh, we, And, and like, this is also a characteristic of the beginning of these cycles is you have technical people who get just swept up in the. Technical bits. And they kind of forget that the only thing that really matters is making good product. And like, that's where we're at right now. Like you have to make good product and, um, you know, you have to have ideas and have conviction, and then you have to go get stuff done really fast so that you can see whether you're full of crap or not about the conviction that you have. And you're not. You have very few patterns at the beginning of a cycle to go snap to. Like, you're not looking at someone else's success and saying, okay, well, like,…

AI assessment note: “Oh god, no. Like, you definitely shouldn't do the latter.”

Q What's the crazy AI prediction that most people would call science fiction that you believe to be true?

A It is already the case that I, I think the frontier models are probably better, uh, like, uh, health diagnosticians than Your average GP is it's a. Good thing to sort of realize and act on, uh, as quickly as possible because we have a whole world of people who have inadequate access to high quality health care, uh, including my own family in rural central Virginia, where, you know, it's just not, not good. Um, and so. Yeah, they're, they're just sort of a bunch of these things, uh, like this where. Yeah, the models are already really good and you've got. Um, you, you basically need the whole world to wake up to the fact that they're good, so that we can go deploy this stuff, and like, Deploy it because the thing that we really care about is the good of the public, uh, not trying to, you know, sustain some status quo.

AI assessment note: “frontier models are probably better, uh, like, uh, health diagnosticians than Your average GP”

Q This is a leading question. Then again, if you think about, and I think you might know, but if you think about those products, who benefits most? Is it startups who are able to integrate new technologies very easily from the bottoms up, starting from nothing? Or is it Microsoft integrating AI into incredible distribution already? Google doing the same. Who benefits most in that respect?

A Again, if you look to past cycles, like you've got to Pretty good mix of where value gets created across startups and new ventures and existing enterprises. And so I think everybody's kind of doing the same thing. Like you're trying to discover the new if you are. If you're a big company like Microsoft with a long tradition and a bunch of successful things already in the market, like the thing that you are trying to do is figure out. What are the things that you already know super well and like which of the customers that you're already serving super well can you do for them with this new set of capabilities that you, uh, you know, that you can provide and. Yeah. Hopefully, you know, like I run among other things, Microsoft research. And so like, I, I also have a charter of like, you know, Hey, can we go try to shine flashlights in places that no one else has shown them before and like try to discover some like super disruptive brand new things. But like, that's kind of the job of the startup ecosystem as well. And I, I also am an angel investor and like, I advise startups and I've worked at startups and. You know, so I, I think it's just really important that you've got lots of people hunting for those interesting new things, and like, I, I have super high conviction on that in the AI platform transition that we're going through right now because, uh, It's impossible for any e…

AI assessment note: “Pretty good mix of where value gets created across startups and new ventures and existing enterprises.”

Q I was listening to your show, as I mentioned earlier, and you push back on the idea of, uh, reaching scaling laws. And this kind of asymptote of, uh, efficiency or effectiveness. When many people suggest that we are hitting scaling more soon. First, why do you think we're not? And that's a ridiculous statement.

A I can very clearly see what we're doing now and like what we're doing next. And I don't see the limit to the scaling laws. Like if you're just sort of thinking about the raw capability of the models and like how well you can condition them to reason over increasingly complicated things like, and I'm sure like There, there at some point will be maybe a limit. Like I, I intuitively feel like there, there must be, there are some people who don't believe that there's a limit that, you know, if you, the limit that human beings have on intelligence is like, you've got so many neurons like packed into your skull and like, you've got about a 20 watt power envelope and like, that's the limit. Some people believe that, you know, if you're You, you have AIs that, like, there, there is no such limit and, like, things, you know, will continue to scale into, like, you know, weird territory. I, I don't really, like, that I don't necessarily believe. I believe we will get to some point where we'll hit a scaling asymptote and, you know, like, there'd just be diminishing marginal returns and, like, it's so expensive that we will decide it's not worth spending that next dollar to, like, make Make this thing one unit smarter because we haven't, you know, figured out how that translates into something that's useful for, uh, like the, the people who are using the tool. Um, I think that point will co…

AI assessment note: “I think that point will come. I don't, I just don't see it yet.”

Q This is a leading question. Then again, if you think about, and I think you might know, but if you think about those products, who benefits most? Is it startups who are able to integrate new technologies very easily from the bottoms up, starting from nothing? Or is it Microsoft integrating AI into incredible distribution already? Google doing the same. Who benefits most in that respect?

A Again, if you look to past cycles, like you've got to Pretty good mix of where value gets created across startups and new ventures and existing enterprises. And so I think everybody's kind of doing the same thing. Like you're trying to discover the new if you are. If you're a big company like Microsoft with a long tradition and a bunch of successful things already in the market, like the thing that you are trying to do is figure out. What are the things that you already know super well and like which of the customers that you're already serving super well can you do for them with this new set of capabilities that you, uh, you know, that you can provide and. Yeah. Hopefully, you know, like I run among other things, Microsoft research. And so like, I, I also have a charter of like, you know, Hey, can we go try to shine flashlights in places that no one else has shown them before and like try to discover some like super disruptive brand new things. But like, that's kind of the job of the startup ecosystem as well. And I, I also am an angel investor and like, I advise startups and I've worked at startups and. You know, so I, I think it's just really important that you've got lots of people hunting for those interesting new things, and like, I, I have super high conviction on that in the AI platform transition that we're going through right now because, uh, It's impossible for any e…

AI assessment note: “Pretty good mix of where value gets created across startups and new ventures and existing enterprises.”

Q Is there anything that many people think about agents that you often hear that you think is wrong?

A I often think that skeptics, uh, about, like, oh, this is, like, hard or impossible, um, like, I, I think they're probably wrong, but, like, I'm not unique in thinking they're wrong, like, you, you just, there are plenty of optimists out there who think that, you know, the technology is going to get more capable, and, you know, it's, like, fine to have skeptics, uh, you know, I don't know what skin in, in the game they have. I have a colleague who wrote this, uh, you know, book called There's No Prize for Pessimism, and there really isn't. Like, I don't know what prize you win for being skeptical about something, uh, if you're not going to go do something about it.

AI assessment note: “I often think that skeptics, uh, about, like, oh, this is, like, hard or impossible”

Q I was listening to your show, as I mentioned earlier, and you push back on the idea of, uh, reaching scaling laws. And this kind of asymptote of, uh, efficiency or effectiveness. When many people suggest that we are hitting scaling more soon. First, why do you think we're not? And that's a ridiculous statement.

A I can very clearly see what we're doing now and like what we're doing next. And I don't see the limit to the scaling laws. Like if you're just sort of thinking about the raw capability of the models and like how well you can condition them to reason over increasingly complicated things like, and I'm sure like There, there at some point will be maybe a limit. Like I, I intuitively feel like there, there must be, there are some people who don't believe that there's a limit that, you know, if you, the limit that human beings have on intelligence is like, you've got so many neurons like packed into your skull and like, you've got about a 20 watt power envelope and like, that's the limit. Some people believe that, you know, if you're You, you have AIs that, like, there, there is no such limit and, like, things, you know, will continue to scale into, like, you know, weird territory. I, I don't really, like, that I don't necessarily believe. I believe we will get to some point where we'll hit a scaling asymptote and, you know, like, there'd just be diminishing marginal returns and, like, it's so expensive that we will decide it's not worth spending that next dollar to, like, make Make this thing one unit smarter because we haven't, you know, figured out how that translates into something that's useful for, uh, like the, the people who are using the tool. Um, I think that point will co…

AI assessment note: “I can very clearly see what we're doing now and like what we're doing next.”

Q You mentioned earlier the centrality of products. Um, taking that into account with the current conversation here, How do we think about whether chat is the right UI for the next kind of paradigm of this product realm? You know, OpenAI and ChatGPT has made it the default. To what extent do you think it is the right default and how we will see that change?

A Like, I think it's a You know, reasonable step in the right direction. Like the thing I've been saying for a few years now is, uh, like, I think one of the most interesting things happening with AI is we we've had one paradigm for using computing devices for effectively 200 years since, uh, Ada Lovelace wrote the first program. So if you want a computing device to go do something for you, you had to be a programmer yourself, which is a pretty, you know, High barrier to entry for a lot of people, or you have to rely on the fact that a programmer has anticipated some need that you might have and like packaged up, uh, like a piece of software into an application that, that you were able to run. And those are the two ways you can get a computing device to do something for you, uh, until now. And so like the, the thing that changes with AI is, It can understand a thing that you want your computing device to go do for you. Uh, and it can figure out a way to go make that thing happen and you don't have to be a programmer. And, you know, it's, it's kind of a profound change because like, it basically means, and like, I don't think this is next year, but it's probably not going to be 10 years. Um, this whole notion that the, That, that you have teams of people who are, who, whose job is to go anticipate a bunch of very granular user needs in some narrow space, and then they're gonna go …

AI assessment note: “I think it's a You know, reasonable step in the right direction.”

Q Is there a role in engineering or product teams that we have today, which you're like, in 20 years time, people will look at it and go, what? You had secretaries who typed out, you know, voice recorded notes from a doctor? What?

A The role of engineers are You're still gonna have to have people who build capability infrastructure, so. You know, make this thing happen in the real world, like, um, you know, provide access to like this, you know, weirdly situated repository of information. Like, you know, it's like just a bunch of capability things that people will need to build, but like the user interface that surfaces, those capabilities will probably be agents. Uh, and, you know, product managers, like I, I don't all don't believe in this, like one agent. For everything sort of theory. I think you'll have a lot of agents. And the reason I think you're going to have a lot of agents is because. Um, your product managers are probably going to have to be domain experts, like people who sort of deeply understand something like medicine or, you know, drug discovery or early round venture investing, or, you know, like, you know, just sort of pick your thing. Uh, and like, you know, they will have to deeply understand the idiosyncrasies of that, that, and they will have to like help set up the feedback loops that help agents that are like assisting people doing those tasks. Like better and better do their job, and so like a little bit like the combination of the product manager and the users of the agents teaching the agents how to be better and better at the things that you're trying to get them to assist you …

AI assessment note: “product managers are probably going to have to be domain experts”

Q Is there a role in engineering or product teams that we have today, which you're like, in 20 years time, people will look at it and go, what? You had secretaries who typed out, you know, voice recorded notes from a doctor? What?

A The role of engineers are You're still gonna have to have people who build capability infrastructure, so. You know, make this thing happen in the real world, like, um, you know, provide access to like this, you know, weirdly situated repository of information. Like, you know, it's like just a bunch of capability things that people will need to build, but like the user interface that surfaces, those capabilities will probably be agents. Uh, and, you know, product managers, like I, I don't all don't believe in this, like one agent. For everything sort of theory. I think you'll have a lot of agents. And the reason I think you're going to have a lot of agents is because. Um, your product managers are probably going to have to be domain experts, like people who sort of deeply understand something like medicine or, you know, drug discovery or early round venture investing, or, you know, like, you know, just sort of pick your thing. Uh, and like, you know, they will have to deeply understand the idiosyncrasies of that, that, and they will have to like help set up the feedback loops that help agents that are like assisting people doing those tasks. Like better and better do their job, and so like a little bit like the combination of the product manager and the users of the agents teaching the agents how to be better and better at the things that you're trying to get them to assist you …

AI assessment note: “You're still gonna have to have people who build capability infrastructure”

Q Is there anything that many people think about agents that you often hear that you think is wrong?

A I often think that skeptics, uh, about, like, oh, this is, like, hard or impossible, um, like, I, I think they're probably wrong, but, like, I'm not unique in thinking they're wrong, like, you, you just, there are plenty of optimists out there who think that, you know, the technology is going to get more capable, and, you know, it's, like, fine to have skeptics, uh, you know, I don't know what skin in, in the game they have. I have a colleague who wrote this, uh, you know, book called There's No Prize for Pessimism, and there really isn't. Like, I don't know what prize you win for being skeptical about something, uh, if you're not going to go do something about it.

AI assessment note: “skeptics, uh, about, like, oh, this is, like, hard or impossible”