The Exchanges

Q think the end state is for the foundational model layer before we move on to the application layer? Like, do they just get bought, you know, you've got a couple of core ones, which is really anthropic, Mistral, who have raised a lot of money, uh, billions, and bluntly will not have the resources to compete continuously in the tens of billions of dollars needed. What happens to this layer?

A I think what happens is, um, all of the tier one clouds will have Um, their own effort that will do well because it has to do well. Um, and they will do whatever it takes to go ensure that they have the capital and data flywheel and talent to go do that. Then I think for the independent companies, and I would say that adept is very different because we're, what we do is we sell an actual end user facing agent to enterprises, which is a very different business model than selling models to, uh, developers. But companies that sell models to developers will either need to effectively be the like first party effort of one of these big clouds, or they have a short window between now and commoditization to build such a big economic flywheel that they can afford to stay independent.

AI assessment note: “companies that sell models to developers will either need to effectively be the like first party effort”

Q And then what's the next wave then? You've had that 20 12 to 20 18. How do you categorize the next wave?

A Well, I think what happened was in 2017 the transformer came out. Um, and I remember, um, I was actually, I was running engineering at OpenAI at the time, and, uh, um, I was working really closely with Ilya, um, and Ilya and I were just sitting around, and, and he was just like, look, this Transformer thing is, is, is, is real. It's gonna be the next most important thing. Let's get all of our teams looking at how we can use this thing. And so the thing that most people in the general public don't know about is we didn't invent Transformer at OpenAI. It was invented at Google. Uh, and, uh, and, and like, but what Transformer did, though, was It was the first time you had a model that was sort of generally applicable to any machine learning task, right? Like back in the day, if you wanted to understand images, you used a convolutional neural network, right? Like Alex net, if you wanted to understand, uh, if you wanted to, to, uh, generate, uh, text, uh, you used RNNs. If you wanted to beat humans at go, you used, uh, you used, uh, uh, like, um, a tree search or RL, right? So you have these, all these different, like different Um, models that you would use to solve problems in AI and then transformer kind of became like the universal model and like the, like the, like base element of AI that came out at that time. And once transformer came out, um, in a weird way, you kind of stop…

AI assessment note: “transformer kind of became like the universal model and like the, like the, like base element”

Q But ChatGPT seems to be the consumer breakthrough that we all waited so many years for. What was the reason for that chasm between Transformer breakthrough and consumer adoption breakthrough with ChatGPT?

A Yeah, no, that's a really good question. It's kind of like ChatGPT was like, Was like the frog that ultimately became boiled, right? Like, Transformer was a huge breakthrough, and every incremental year from 2017 to when ChatGPT came out, um, language models just got a little better, and a little better, and a little better. Like, I remember, um, Alec Radford, um, and a couple others, and I did GPT-II, right? And GPT-II came out in, I think, 2019. I just remembered, um, you know, this thing, this, finally you had this, like, pretty smart generalist Model that you could just say, hey, like, write me a newspaper article about insert celebrity being arrested in L.A., and it would just do a perfect job and be like, oh, they were in, like, they were in, like, the Neiman Marcus store, et cetera. Um, I thought that was so much fun, and, um, and, um, but, like, the thing is, like, two things had to happen. One, the models weren't getting increasingly smart, but there's, like, a minimum viable smartness where you're, like, damn, this is a compelling experience. And the second thing was it needed to be packaged up in a way that consumers could play with. So if you go look at the lag, right, ChatGPT was really just GPT-III with, uh, instruction. It was basically like more chat tuning, but GPT-III API came out, I think, over a year before ChatGPT did, but only developers could play with it…

AI assessment note: “two things had to happen. One, the models... minimum viable smartness... second thing was it needed to be packaged”

Q Is it easy? Like when you think about Nvidia and what they do, is it easy for them to be commoditized? Is there not such sophistication that actually it's incredibly hard for these people to move into the chip player and take that lunch?

A Okay. I actually think we're saying the same thing. It is incredibly hard. Like Nvidia is killing it. It is incredibly hard, but it is possible. And if the economic returns are high enough, people will do it. Right. So I just think Google TPU is a great example. Like the, the, um, Um, I am, I'm, I'm a massive NVIDIA fanboy. Jensen is incredible, like, brilliant guy. Uh, I, I think, um, and I think NVIDIA has, like, executed so well here. I think we also have to give props, though, to, like, I think the TPU team when I was at Google was, like, sub-five hundred people, um, and their budget was a shoestring budget, and yet somehow every generation they taped out, um, quite good chips that were then used to train Gemini and, and Palm and are used by third parties now and all of that stuff, and Like, there is such a strong will to ensure that Google has its own first party chip, um, that I think that's like the, the, the counter example to like the, uh, uh, the perpetual chip dominance.

AI assessment note: “It is incredibly hard, but it is possible.”

Q I always swear This is what RPA was. So can you help me understand the distinction between that traditional RPA, which is what we've seen with UiPath and this new era of agent that we see today?

A Yeah, totally. I mean, this is a, this is, this is really, this is a good question. It's actually a question used to cause me a lot of heartburn because I found it so hard to explain to people why agents were going to be different than RPA. Best analogy I've got is, um, RPA is very useful for high volume tasks that always look the same. So, um, example, uh, the analogy that I would give would be like RPA is a little bit like, um, a little bit like, you know, when you go to a factory floor and there's robots roaming around everywhere. What those robots do is there's like a literally yellow line painted on the floor and the robots like follow that line. They go from cell to cell and station to station and they pick up stuff. But what agents are, are agents are meant to be, agents are meant to be constantly thinking and reevaluating and planning at every step to solve your goal. And it's much more like full self-driving. The difference in utility between those two things is, is, is fairly large. Of course, there's many areas where you don't want, uh, something that can have variability and therefore you should use RPA. But I think the majority, like, I just think in five to 10 years, people are gonna use their computers by giving them high-level goals, right?

AI assessment note: “agents are meant to be constantly thinking and reevaluating and planning at every step”

Q think the end state is for the foundational model layer before we move on to the application layer? Like, do they just get bought, you know, you've got a couple of core ones, which is really anthropic, Mistral, who have raised a lot of money, uh, billions, and bluntly will not have the resources to compete continuously in the tens of billions of dollars needed. What happens to this layer?

A I think what happens is, um, all of the tier one clouds will have Um, their own effort that will do well because it has to do well. Um, and they will do whatever it takes to go ensure that they have the capital and data flywheel and talent to go do that. Then I think for the independent companies, and I would say that adept is very different because we're, what we do is we sell an actual end user facing agent to enterprises, which is a very different business model than selling models to, uh, developers. But companies that sell models to developers will either need to effectively be the like first party effort of one of these big clouds, or they have a short window between now and commoditization to build such a big economic flywheel that they can afford to stay independent.

AI assessment note: “will either need to effectively be the like first party effort of one of these big clouds”

Q Maybe it's the pain and the lack of anesthetic, or maybe it's just my naivety, but I'm not, not afraid to ask this one. Would you say Adept is a foundation model company? Or would you say that actually you are not? How do you think about the positioning?

A We're actually just focused. We're really, really focused on this particular problem that we're trying to solve. Like we're trying to build an AI agent that you can delegate arbitrary work tasks to, right? And so then everything we do stems from that. So what we are not doing is we are not trying to just train foundation models to sell them to other people. What we're doing is we're building like a very vertically integrated stack Going back to our previous discussion of where will vertical integration happen versus not. I do think that in the agent space, it's extremely important that you own the entire stack from what is the end user interface. I think like we were talking about the Apple example earlier, owning the interface gives you tremendous leverage in this era of AI to how do you, uh, make agents that are reliable enough to be used at work all the way down to what needs to happen to the foundation modeling layer to enable this whole end to end system to be maximally performant. That's what we do is this vertical slice.

AI assessment note: “what we are not doing is we are not trying to just train foundation models”

Q I mean, I love that in terms of also Uh, LLMs not being products in themselves. You mentioned five to seven core providers winning. What will separate those that win versus those that don't? Is it purely a game of resources and cash?

A I think it's a game of how much you existentially need to win. I think every cloud provider, um, sorry, every tier one cloud provider existentially needs to win here, right? Like, let's just look at, like, let's look at the dynamics involved. It's, it's one where, you know, as these models get smarter and smarter, They kind of become the base computing primitive. Like today, the base computing primitive is like nodes on EC two or like, uh, storage, right? But in the future, when more and more software is just like the logic of software is actually just handled by a, by an LLM, nobody cares anymore about what the base computing primitive is. All you need to do is access these models and compose these models to go solve things for customers. So then whoever controls the model layer controls Uh, all of the underlying compute. And so, like, right now what's happening, right, is like, is like, um, if you don't have an offering here, then, uh, that is state of the art, then you're just gonna be cut out of this particular game. I think this is also actually an area where I think it's really important for companies like Nvidia to go up the stack, right? Like Nvidia clearly killing it right now on, on, on chips. But what's happening is every one of the major clouds is working on, um, uh, And every major LLM provider is working on a strategy to, to, um, have their in-house chips because …

AI assessment note: “I think it's a game of how much you existentially need to win.”

Q You're writing in pretty morselm. On why that does not happen? Why agents are not that in five years time? What is the most probable reason why that does not happen?

A I think one way it won't happen is that incumbents have so much, like, fundamentally, like, agents are a reframing of where, of, like, how software is bundled, right? Like, today we've, we, we, we, we bundle software in these, like, functional ways, right? Like, you've got You've got Notion or Google Docs for, for your, for your docs, and then you've got Salesforce for sales, and then you've got, like, um, you've got, like, Workday for HR and all of this stuff, right? But the, the work that we do fundamentally spans all of these different domains, and agents should bridge those domains, otherwise you can't become a, a, a, you can't become a higher level, uh, thing. So if we're locked into, like, end walled gardens by incumbents, then that vision will not happen.

AI assessment note: “if we're locked into, like, end walled gardens by incumbents, then that vision will not happen.”

Q Is it like reinforcement learning of its own data sets? Like where it just continuously repeats the same things until it gets it right? Is that a good understanding of it?

A That is a good understanding of it. So I think, um, I think a good way to think about it is historically for the last couple years as we scaled up LLMs, we've just been doing more unsupervised learning. Right? Like, um, get more data, more, more, more, um, more smart, uh, journalists writing articles, feed it in there, and that makes it smarter. But the problem is, like, a model trained that way is only as good as the smartest data in the training set. Like, it cannot discover new knowledge, because its job, the way the models are trained, is to do what a human would do in that situation. But the underlying thing is, if you want to go solve, like, really big problems, like solve, like, prove unproven math Theorems or like be able to like help you solve a creative problem at work. Those problems are by definition things that are not in the training set because it's a, it's a, it's a, it's either like a superhuman thing or it's a novel situation.

AI assessment note: “That is a good understanding of it.”

Q We mentioned ChatGPT there. Before we dive into the meat of the show, I do just have to ask, you mentioned your time with OpenAI. It's such a transformative place to be. You mentioned working with Ilya there. What's one or two of your biggest takeaways from OpenAI that really informed how you think about building a debt?

A Well, the first one actually is, and going back to the, like, eras of AI discussion we were having, right, the, um, what OpenAI realized before basically everybody but DeepMind was that the next phase of AI after Transformer was not going to be about, you know, research paper writing. It was going to be about let's choose a major unsolved Scientific problem and just try to solve it. Right. And so like that led us to go build a culture instead of like loose collections of federations of researchers. Let's put a giant team around. How do you solve like robot hand control? Let's put a giant team around beating humans at like one of the most popular video games on the planet. Right. Let's put a giant team around scaling GPT until this thing is like a generalist reasoning and chat engine. Like that's just a totally different framework from like this very academic curiosity driven research. And I think that that's the right framework. And, uh, and I think it's a big part of how we build the depth now as well.

AI assessment note: “I think it's a big part of how we build the depth now as well.”

Q Is it easy? Like when you think about Nvidia and what they do, is it easy for them to be commoditized? Is there not such sophistication that actually it's incredibly hard for these people to move into the chip player and take that lunch?

A Okay. I actually think we're saying the same thing. It is incredibly hard. Like Nvidia is killing it. It is incredibly hard, but it is possible. And if the economic returns are high enough, people will do it. Right. So I just think Google TPU is a great example. Like the, the, um, Um, I am, I'm, I'm a massive NVIDIA fanboy. Jensen is incredible, like, brilliant guy. Uh, I, I think, um, and I think NVIDIA has, like, executed so well here. I think we also have to give props, though, to, like, I think the TPU team when I was at Google was, like, sub-five hundred people, um, and their budget was a shoestring budget, and yet somehow every generation they taped out, um, quite good chips that were then used to train Gemini and, and Palm and are used by third parties now and all of that stuff, and Like, there is such a strong will to ensure that Google has its own first party chip, um, that I think that's like the, the, the counter example to like the, uh, uh, the perpetual chip dominance.

AI assessment note: “It is incredibly hard, but it is possible.”

Q and not the tools. It's like the, the hottest statement, um, and the end of price per seat, and we're all moving to a consumption-based pricing model. Do you agree with that statement? Do you think we're all slightly over emphasizing, like, the end of price per seat, and how do you feel about this kind of fundamental shift in, in business model and pricing that AI could bring about?

A I think in places we're definitely going to see that become true, but I actually think in knowledge work, That's, I think in knowledge work, the most valuable things to do will not be priced that way. And, and, and here's why. Um, I think that like the definition of price per work assumes repetitiveness, commoditization, cookie cutter, you know, like no creativity. Right. Um, but the way that, like, I think what these AI systems are going to do, especially AI agents are going to do, we are basically going to like give people The ability to go, to go do new things and have way more leverage on their time and like give them more opportunities to be creative. And so then ultimately what we're building is like a co-pilot or a teammate and co-pilots and teammates don't charge you a price per work. They like, like you really pay them based on their ability to augment your ability to go, to go do new things. Right.

AI assessment note: “in knowledge work, the most valuable things to do will not be priced that way”

Q Why are RPA players not best placed to provide an agent solution to existing customers?

A I think it's just really fundamentally disruptive to their business model. Like, um, the way that, um, the way that a big corporation, um, uses UiPath, right, is like, um, is like, there's a, there's a, there's a big, like, big plans around a process transformation that needs to be done. Sometimes like a, like an Accenture or something comes in, And then maps out what the processes are like sometimes with a process discovery, um, uh, thing. And then our PA engineers go and build those workflows. And then six to nine months later, you hit play on this thing that then automates some invoice processing every night or something like that. Right? Like this new model of like, you just put an agent in there and the agent observes what the end user does to go do that job. And then that becomes like a thing that you can then just invoke with natural language. It's like really disruptive to the business model. Um, and I think that, um, I think that the best way to, uh, to, to run circles around incumbents is to do something that has a different business model than what they have.

AI assessment note: “I think it's just really fundamentally disruptive to their business model.”

Q come out as being true. Um, I had some famous people call me an idiot, which actually made me quite pleased when those revenue numbers were revealed. How do you feel about implementation providers, AI services providers, Being bigger than the actual providers in the next five years. Do you think that it's right that the biggest players to come out of this cycle will be the AI services providers?

A I don't think so because I think the third bucket of the third bucket of like economic upside is still early. And I think that bucket is like, um, is the companies that then turn the use cases that have product market fit into repeatable products. Right now, right, imagine you're, you're, you're very large company X, right, and you need capability Y, and then you've got the base model over here, right, that's pretty darn smart, GPT-FOR, or, or Gemini, or whatever, right, um, and there's a giant gulf in the middle. Like, the first, I feel like in, um, in, in every one of these cases, the first people that go fill that gulf are, like, sort of consulting-y service providers, right, but then the moment that gulf starts getting filled, you start seeing, ah, ok, like, this is the really useful thing for an enterprise, Then people just go productize that thing, and then that becomes a startup. And so then that becomes eventually a company that's a conduit between the base intelligence and the customer. So today that might be true for services, but I feel like a lot of these things will be turned into generalizable products. And then when they do, those companies will then be the real economic winners.

AI assessment note: “I don't think so because I think the third bucket”

Q Why is it just starting to be tapped now? And what does that progression look like of its own development?

A Everything's an S curve, right? So the giant model scaling base model scaling S curve for the last couple of years, we were here. We were at the sharpest point of improvement, right? You just, you, um, you, everybody could, you know, you could double the cost of your model, a hundred million dollars to two hundred million dollars, and that would be the fastest and easiest way to deliver a smarter thing to, to, to, To the world. And now if you're getting the billion dollar training runs and two billion dollar training runs and four billion dollar training runs, it's really freaking hard to go get more money to go, uh, to go make the base thing bigger. And so because of that, now the critical path for model improvement is, is, uh, is shifting over to this, uh, to this, um, uh, sort of, uh, broader sort of simulation slash, um, synthetic data slash like RL loop sort of path. Uh, I think it's just a natural consequence of the fact that it's so expensive to just keep scaling.

AI assessment note: “it's just a natural consequence of the fact that it's so expensive to just keep scaling”

Q I mean, I love that in terms of also Uh, LLMs not being products in themselves. You mentioned five to seven core providers winning. What will separate those that win versus those that don't? Is it purely a game of resources and cash?

A I think it's a game of how much you existentially need to win. I think every cloud provider, um, sorry, every tier one cloud provider existentially needs to win here, right? Like, let's just look at, like, let's look at the dynamics involved. It's, it's one where, you know, as these models get smarter and smarter, They kind of become the base computing primitive. Like today, the base computing primitive is like nodes on EC two or like, uh, storage, right? But in the future, when more and more software is just like the logic of software is actually just handled by a, by an LLM, nobody cares anymore about what the base computing primitive is. All you need to do is access these models and compose these models to go solve things for customers. So then whoever controls the model layer controls Uh, all of the underlying compute. And so, like, right now what's happening, right, is like, is like, um, if you don't have an offering here, then, uh, that is state of the art, then you're just gonna be cut out of this particular game. I think this is also actually an area where I think it's really important for companies like Nvidia to go up the stack, right? Like Nvidia clearly killing it right now on, on, on chips. But what's happening is every one of the major clouds is working on, um, uh, And every major LLM provider is working on a strategy to, to, um, have their in-house chips because …

AI assessment note: “I think it's a game of how much you existentially need to win.”

Q Maybe it's the pain and the lack of anesthetic, or maybe it's just my naivety, but I'm not, not afraid to ask this one. Would you say Adept is a foundation model company? Or would you say that actually you are not? How do you think about the positioning?

A We're actually just focused. We're really, really focused on this particular problem that we're trying to solve. Like we're trying to build an AI agent that you can delegate arbitrary work tasks to, right? And so then everything we do stems from that. So what we are not doing is we are not trying to just train foundation models to sell them to other people. What we're doing is we're building like a very vertically integrated stack Going back to our previous discussion of where will vertical integration happen versus not. I do think that in the agent space, it's extremely important that you own the entire stack from what is the end user interface. I think like we were talking about the Apple example earlier, owning the interface gives you tremendous leverage in this era of AI to how do you, uh, make agents that are reliable enough to be used at work all the way down to what needs to happen to the foundation modeling layer to enable this whole end to end system to be maximally performant. That's what we do is this vertical slice.

AI assessment note: “what we are not doing is we are not trying to just train foundation models”

Q You mentioned kind of AGI being kind of infinity there in people's minds. You said before to me that the last step is human computer interaction, and that's the last ingredient to AGI. Before we do a quick fire, what did you mean by that? I didn't get that one either.

A I personally find a world in which sort of increasingly generally intelligent systems run around with their own agency and goals, uh, and, uh, and, um, not involve, um, Not involve what, what, what humans most care about to be not a world that I really want to live in. And this goes back to the, what you were saying about like selling AI by work versus as a, as a, as a software tool, right? Like I, I would much rather live in a world where we have sort of these like, like AI teammates, um, and assistants that we interact with instead. And then I think the question becomes, um, the, then the question becomes, how do you find the right interface Between smarter and smarter AI systems and people and how that interface is defined actually changes a lot about what training data you collect. How can humans align these systems towards the preferences of what humans want? Also, ultimately, like how these models are even built and what their architectures are. And so in a weird way, like the way the field is moving is let's make models smarter and then let's make use cases starter and then let's go put them in people's hands and then let's figure out what this means for people. Like it's kind of this waterfall sequential method. Which I don't think is a very good way to develop the technology. I think we should start back from ultimately how did, how should humans use these things and t…

AI assessment note: “how do you find the right interface Between smarter and smarter AI systems and people”

Q We mentioned ChatGPT there. Before we dive into the meat of the show, I do just have to ask, you mentioned your time with OpenAI. It's such a transformative place to be. You mentioned working with Ilya there. What's one or two of your biggest takeaways from OpenAI that really informed how you think about building a debt?

A Well, the first one actually is, and going back to the, like, eras of AI discussion we were having, right, the, um, what OpenAI realized before basically everybody but DeepMind was that the next phase of AI after Transformer was not going to be about, you know, research paper writing. It was going to be about let's choose a major unsolved Scientific problem and just try to solve it. Right. And so like that led us to go build a culture instead of like loose collections of federations of researchers. Let's put a giant team around. How do you solve like robot hand control? Let's put a giant team around beating humans at like one of the most popular video games on the planet. Right. Let's put a giant team around scaling GPT until this thing is like a generalist reasoning and chat engine. Like that's just a totally different framework from like this very academic curiosity driven research. And I think that that's the right framework. And, uh, and I think it's a big part of how we build the depth now as well.

AI assessment note: “I think it's a big part of how we build the depth now as well.”

Q When we think about, like, actually improvements in models, they lead to improvements in performance, and I kind of think there's three ways of doing that, one of which is like a breakthrough in reasoning. How do you think about the likelihood of a breakthrough in reasoning? What's required for that? And whether that is a reasonable expectation?

A Reasoning is one of the problems in the field right now that I think, um, a bunch of us sort of have similar ideas for how to solve. But it actually requires some new research to be done. So, um, in a, in a weird way, working in AI is pretty funny these days because, because, um, the giant model scaling problem is so known and it's really a function of resources. And so if you, and so you kind of don't feel like you need to be a genius to go make new parts and just pure model scaling. But I think pure model scaling does not deliver solutions to reasoning. To me, the definition of reasoning is Being able to, uh, is being able to like compose existing thoughts to discover some new thought, right? Um, and I think to go do that, that's not something that's trained into the capabilities of LLMs by simply asking it to regurgitate the internet's worth of data. The way we're going to solve reasoning is back to what we were talking about earlier of like taking theorem proving as an example. You want to give the model access to a theorem proving environment and have it try things. Um, in the same way that like, you know, a human mathematician would sit down and be like, well, you know, here's the things I know to be true about the world. How do I compose them such that I can prove the thing that I want to prove?

AI assessment note: “The way we're going to solve reasoning is back to what we were talking about earlier”

Q Can you just unpack that for me? Keep going with that thought.

A I love the topic of chips. We can stay here for a while. It's just, it's so much fun. It's like the most interesting thing, uh, uh, that's happened in some time for that industry. Like the, um, uh, so, so we were just talking a minute ago about how How important it is, right, for, um, uh, for model makers to control their chips, because that way, you know, if it really is a scale and resources game, if company A, let's say choose Google with TPU, TPUs are great, right, um, with TPU has a 20% cost advantage compared to company B using chip Y, right, then Google will just be able to, just the better cost of model training will let them go bigger, let them invest more in Post training tricks like the ones we were talking about earlier and have an advantage. And so then company, company B is like going to be really pressured to go find some way to go, go, go do that themselves. Similarly, if you're a chip maker, um, if, uh, if you become like, it's just too easy to be, to be commoditized by these in-house efforts if you don't also own something at the model layer.

AI assessment note: “for model makers to control their chips, because that way, you know, if it really is a scale”

Q Why are RPA players not best placed to provide an agent solution to existing customers?

A I think it's just really fundamentally disruptive to their business model. Like, um, the way that, um, the way that a big corporation, um, uses UiPath, right, is like, um, is like, there's a, there's a, there's a big, like, big plans around a process transformation that needs to be done. Sometimes like a, like an Accenture or something comes in, And then maps out what the processes are like sometimes with a process discovery, um, uh, thing. And then our PA engineers go and build those workflows. And then six to nine months later, you hit play on this thing that then automates some invoice processing every night or something like that. Right? Like this new model of like, you just put an agent in there and the agent observes what the end user does to go do that job. And then that becomes like a thing that you can then just invoke with natural language. It's like really disruptive to the business model. Um, and I think that, um, I think that the best way to, uh, to, to run circles around incumbents is to do something that has a different business model than what they have.

AI assessment note: “I think it's just really fundamentally disruptive to their business model.”

Q and actually he states his kind of concern that AI will replicate autonomous driving in the way that we got so excited about 10 years ago. Everyone's gonna be unemployed, eight million truck drivers, but my question to you is, are we gonna see that similar plateauing Where for 10 years actually, kind of autonomous cars didn't feel like it was progressing. Do you, how do you think about that?

A I've never worked in self-driving, um, but if I, I'm gonna, I'm gonna apply a mental model, and you tell me, you, you tell me if it, if it lines up. I feel like in self-driving what happened was, there was an aha moment where, you know, you could get the thing to work at all, and then you're like, okay, well now it works 60% of the time. How do we get this to 99.99999% of the time? And every day you show up to work and you just play whack-a-mole on what's not working. Uh, and you just like hope and pray that this converges to that like 99.99999 thing. That's not true for AI right now. That's, I'm sorry, that's not true for, uh, specifically what I'm about to say is only applicable to building smarter and smarter models and agentic systems that ultimately help you do work. That's, that's the thing that I'm trying to talk about. Like, Um, for building that, that's not how, that's not how the, the underlying dynamics are right now. Like every day we go to work and there's like actually brand new scientific things we want to try that just dramatically improve the performance of the model. And, um, some of those bets don't work and some of those bets really work. Um, I think like the reasoning that we talked about earlier is an example of one. I think another example of one is like this, like universal multimodality that GPT-IV-O is. Like those, those breakthroughs are like visible.…

AI assessment note: “what's going to prevent this from just being a hype cycle that falls flat”

Q come out as being true. Um, I had some famous people call me an idiot, which actually made me quite pleased when those revenue numbers were revealed. How do you feel about implementation providers, AI services providers, Being bigger than the actual providers in the next five years. Do you think that it's right that the biggest players to come out of this cycle will be the AI services providers?

A I don't think so because I think the third bucket of the third bucket of like economic upside is still early. And I think that bucket is like, um, is the companies that then turn the use cases that have product market fit into repeatable products. Right now, right, imagine you're, you're, you're very large company X, right, and you need capability Y, and then you've got the base model over here, right, that's pretty darn smart, GPT-FOR, or, or Gemini, or whatever, right, um, and there's a giant gulf in the middle. Like, the first, I feel like in, um, in, in every one of these cases, the first people that go fill that gulf are, like, sort of consulting-y service providers, right, but then the moment that gulf starts getting filled, you start seeing, ah, ok, like, this is the really useful thing for an enterprise, Then people just go productize that thing, and then that becomes a startup. And so then that becomes eventually a company that's a conduit between the base intelligence and the customer. So today that might be true for services, but I feel like a lot of these things will be turned into generalizable products. And then when they do, those companies will then be the real economic winners.

AI assessment note: “I don't think so because I think the third bucket of like economic upside”

Q Why has no one been able to solve memory? People often talk about this, and respectfully, it seems a confusing one to me, because it's like, computers have memory anyway. Why, why in AI is memory such a challenge?

A You can kind of think about memory as being two different things, right? You kind of have short-term working memory, and then you have long-term memory. I think people have made really good progress on short-term working memory, right? Like, um, if you could look at Gemini, Gemini's context length is like a million, it might even be more now, I actually don't quite remember, like a, like a million tokens long, which is so cool, as you can feed it, like, Giant snippets of video and be like, hey, like, write me a step-by-step of, like, every, everything the person cooking on this, uh, on this, in this particular video did, and it'll do it. Like, that stuff is insane. Like, that's making good progress, and the reason that's been hard is for computational reasons, um, but I think this, this sort of longer-term memory problem, this goes back to, like, another thing that I believe, and that, that's why I'm excited, I'm, I'm, I'm slightly less excited about model building, slightly more excited about application developers, because The underlying thing that everyone's realizing now is that LLMs themselves are not a product. Like, an actual product is this entire software system that uses LLMs in it. So for example, like, um, uh, what we should be doing is we should be finding ways in which, uh, in which, like, end application builders can be themselves responsible for, uh, for how to …

AI assessment note: “You can kind of think about memory as being two different things, right?”

Q What does that do to the org structures of teams, David, do you think? Does this mean much, much smaller companies? Does this mean, how do you think that actually plays out?

A I think the main way it plays out is, um, actually, this is something I'm gonna steal from, um, our angel investor, Scott Belsky, who, uh, has just thought about this so much. Um, he always calls it like this collapsing the talent stack thing, and the idea is basically that, um, uh, or this is my interpretation on it, like, projects and teams where the same person is simultaneously the PM and the designer and or the engineer or the go-to-market person or the marketer or whatever, Like, the more that, like, those different skill sets are smushed in the same person, the faster that thing moves and the more effective the thing becomes. So I think what it's gonna do is it's gonna make humans at work much more like generalists, and it's gonna have, like, causes to create larger and larger, oh, sorry, like, uh, uh, or giving people sort of, like, larger and larger scope over various different, like, areas that are different functions today, while they ultimately supervise, like, a cohort of, like, AI co-pilots that are the specialists.

AI assessment note: “it's gonna make humans at work much more like generalists”

Q What does that mean? What pure bottoms up good basic research is?

A Ah, yeah. So, so I have this worldview, um, of, of AI progress as being a part of a couple of different phases, right? And I, I like to think about pre-twenty-twelve as basically being prehistory. Of course, um, all of the, uh, OGs in the field were probably not like that if I characterized it that way, but before twenty-twelve, like most of the things we tried just didn't really work, right? Like you, you had things like, um, uh, a sheep being identified as cats and dogs and, uh, Chatbots that, um, barely said anything coherent, et cetera. But I think, like, there was a, uh, there was a period between twenty-twelve, like, twenty-seventeen or twenty-eighteen, where deep learning went from something that, um, people didn't believe in to being, like, the dominant paradigm in the field, right? And so during that twenty-twelve, twenty-eighteen era, the way people made progress, um, what I mean by bottom-up basic research is you hire the most brilliant scientists, they come to work every day with, like, No, um, near-term objective they're being held accountable to. Um, and they just work together, and they think about, you know, hmm, like, I wonder what it'd be like if we could solve this, like, open, uh, technical problem in AI. Like, how do we go, um, how do we create a model that better understands how to generate images? And they just go work on that of their own curiosity and d…

AI assessment note: “what I mean by bottom-up basic research is you hire the most brilliant scientists”

Q But ChatGPT seems to be the consumer breakthrough that we all waited so many years for. What was the reason for that chasm between Transformer breakthrough and consumer adoption breakthrough with ChatGPT?

A Yeah, no, that's a really good question. It's kind of like ChatGPT was like, Was like the frog that ultimately became boiled, right? Like, Transformer was a huge breakthrough, and every incremental year from 2017 to when ChatGPT came out, um, language models just got a little better, and a little better, and a little better. Like, I remember, um, Alec Radford, um, and a couple others, and I did GPT-II, right? And GPT-II came out in, I think, 2019. I just remembered, um, you know, this thing, this, finally you had this, like, pretty smart generalist Model that you could just say, hey, like, write me a newspaper article about insert celebrity being arrested in L.A., and it would just do a perfect job and be like, oh, they were in, like, they were in, like, the Neiman Marcus store, et cetera. Um, I thought that was so much fun, and, um, and, um, but, like, the thing is, like, two things had to happen. One, the models weren't getting increasingly smart, but there's, like, a minimum viable smartness where you're, like, damn, this is a compelling experience. And the second thing was it needed to be packaged up in a way that consumers could play with. So if you go look at the lag, right, ChatGPT was really just GPT-III with, uh, instruction. It was basically like more chat tuning, but GPT-III API came out, I think, over a year before ChatGPT did, but only developers could play with it…

AI assessment note: “The packaging and the intelligence had to exist in order for”

Q Can you just unpack that for me? Keep going with that thought.

A I love the topic of chips. We can stay here for a while. It's just, it's so much fun. It's like the most interesting thing, uh, uh, that's happened in some time for that industry. Like the, um, uh, so, so we were just talking a minute ago about how How important it is, right, for, um, uh, for model makers to control their chips, because that way, you know, if it really is a scale and resources game, if company A, let's say choose Google with TPU, TPUs are great, right, um, with TPU has a 20% cost advantage compared to company B using chip Y, right, then Google will just be able to, just the better cost of model training will let them go bigger, let them invest more in Post training tricks like the ones we were talking about earlier and have an advantage. And so then company, company B is like going to be really pressured to go find some way to go, go, go do that themselves. Similarly, if you're a chip maker, um, if, uh, if you become like, it's just too easy to be, to be commoditized by these in-house efforts if you don't also own something at the model layer.

AI assessment note: “for model makers to control their chips, because that way”