The Exchanges

Q I told you, I tweeted before, like, we should skip the founding stories because there are many, many great times you've told it before, but I want to dive straight in, and I want to ask you the question of when we look at model performance today, let's just start high level. Do you think we're seeing a case of diminishing returns where more compute doesn't lead to better performance?

A Yeah, I think it's pretty fascinating. I mean, I think there's been, um, this is especially coming up now, where GPT-IV, uh, you know, OpenAI has had GPT-IV since fall of twenty-twenty-two. Uh, and since that time frame, we haven't yet seen a new base model or a new, a new model that's, you know, um, jaw-droppingly better than GPT-IV. You know, we haven't seen the GPT-IV or the GPT-V or, you know, the other labs haven't yet come out with models that are, You know, leagues and leagues better than GPT-IV, despite just, like, way, way more compute expenditure. You know, since, since when ChatGPT came out, you know, you can look at the graph of NVIDIA's revenue, it just inflects. It's just like, it just goes straight up after GPT-IV came out, and it goes from, you know, I think the NVIDIA's data center revenue, they were doing, uh, roughly about five billion a quarter, and then it shoots up to now it's, you know, north of twenty billion. Dollars a quarter. So there's been, you know, tens of billions going to a hundred, more than a hundred billion of spend on high-end NVIDIA GPUs all in the same time frame. We haven't yet seen the big breakthrough since GPT-IV, which actually that model was, came out before this huge inflection in NVIDIA expenditure. So, uh, I think overall it's very, it's this interesting thing where we're seeing investment into compute go up dramatically, go up ex…

AI assessment note: “We haven't yet seen the big breakthrough since GPT-IV”

Q I thought that is absolute shit. Uh, and I think when you look at like data provisions and what the Chinese government will be willing to do in terms of In terms of data access and data provisions and regulation, I, I think if they are two years behind, that will very quickly catch up. How do you see China being two years behind, and do you agree with that?

A Two years ago, there were probably more than two years behind. So, um, when, you know, when OpenAI first produced GPT-IV in the lab, uh, you know, China were nowhere near that. But just even the past few months, uh, there was a Chinese company, zero one, zero one dot AI, Uh, that produced, uh, a model YI-large, uh, YI-large, that is now the, uh, one of the best models in the world. I think it's just behind, so it's behind GPT-IV-O and, uh, Gemini Claude III Opus, and it's the next model right behind that in the leaderboards. So it's one of the best models in the world. Um, so we've already seen them meaningfully catch up. They are like, you know, Chinese, Chinese LLM and AI capabilities are, I would say right now, pretty close to neck and neck with us capabilities. And I think if you plot the path ahead based on everything we've talked about with data, I mean, I think they're gonna, they have a clear shot at racing forward and racing ahead of us. And I think it comes down to at its core, the, that the CCP system is incredibly good at Um, at taking, uh, very aggressive centralized action and centralized industrial policy to drive forward critical industries. And what we've seen even in the past few years on, or the past few decades, frankly, on solar, um, uh, how China has, or the CCP has been able to make, um, uh, take industrial policy to the point of like being by and large t…

AI assessment note: “Chinese LLM and AI capabilities are, I would say right now, pretty close to neck and neck”

Q Why are we not going to get there from internet data? When we think about effective agents, and when we think about software doing the work, not just selling the tools, as I think Sarah Townville said quite well before, why is existing data, ah, not equipped to do that transition from tools to work?

A Yeah. So the, um, the simple answer is like a lot of the, uh, a lot of the thought process and a lot of the thinking that humans go through when they are doing more complex tasks that doesn't get written down on the internet. So for example, if I'm a fraud analyst, um, inside a large bank, you know, I go through, my job is like understanding with, based on a set of transactions that seems suspicious, um, whether or not It's a fraudulent, ah, it's a fraudulent transaction, and I need to analyze all sorts of different pieces of data, and use my deduction, and use all my, like, sort of, ah, human intelligence to make that decision, and that process that I go through, it's not like every, like, I'm writing down step by step, like, oh, I looked at this piece of data, and I looked at this piece of data, and then, based on that, I deduce this, and I'm not writing all that down on the internet to later be crawled, ah, by these models. So, one way to think about is, like, all of the reasoning and thinking That is powering the economy today. None of that gets written down on the internet. And so if you just train on the internet, the model has no ability to learn from all of that.

AI assessment note: “a lot of the thought process and a lot of the thinking that humans go through”

Q I respect you, do your best work, and you just have to identify which camp someone's in, and then hopefully, if their skills are there, they should operate to their best. I wish I'd known that when I started, and I didn't, and I just Tried to act out of fear for everyone there. What do you know now that you wish you'd known, and where did you fuck up?

A The biggest one was, um, was actually, you know, in the same era of like, 20, 20, 21, was thinking that hypergrowth as a company meant that you had to hypergrow your team. And so, in those few years, we did what a lot of tech companies did. We, like, doubled, tripled the team year on year, like the, the, you know, from, uh, in 20, 20, we were about a 150 people. By the end of 20, 22, Uh, we were over 700. And it was just this, it was this insane amount of hiring and this incredible amount of, of, of hyper growth as a team. And what I found out is when you hire that quickly, it is impossible to, to do what we've just been talking about, which is maintaining this high bar and maintaining this, this sort of feeling of excellence within the team.

AI assessment note: “thinking that hypergrowth as a company meant that you had to hypergrow your team.”

Q I thought that is absolute shit. Uh, and I think when you look at like data provisions and what the Chinese government will be willing to do in terms of In terms of data access and data provisions and regulation, I, I think if they are two years behind, that will very quickly catch up. How do you see China being two years behind, and do you agree with that?

A Two years ago, there were probably more than two years behind. So, um, when, you know, when OpenAI first produced GPT-IV in the lab, uh, you know, China were nowhere near that. But just even the past few months, uh, there was a Chinese company, zero one, zero one dot AI, Uh, that produced, uh, a model YI-large, uh, YI-large, that is now the, uh, one of the best models in the world. I think it's just behind, so it's behind GPT-IV-O and, uh, Gemini Claude III Opus, and it's the next model right behind that in the leaderboards. So it's one of the best models in the world. Um, so we've already seen them meaningfully catch up. They are like, you know, Chinese, Chinese LLM and AI capabilities are, I would say right now, pretty close to neck and neck with us capabilities. And I think if you plot the path ahead based on everything we've talked about with data, I mean, I think they're gonna, they have a clear shot at racing forward and racing ahead of us. And I think it comes down to at its core, the, that the CCP system is incredibly good at Um, at taking, uh, very aggressive centralized action and centralized industrial policy to drive forward critical industries. And what we've seen even in the past few years on, or the past few decades, frankly, on solar, um, uh, how China has, or the CCP has been able to make, um, uh, take industrial policy to the point of like being by and large t…

AI assessment note: “Chinese LLM and AI capabilities are, I would say right now, pretty close to neck and neck”

Q Given that concern, should we not have closed systems? Obviously, open systems have a lot of benefits, but the challenge with open systems is anyone can use them, and that means that Russia can use them, China can use them, and everyone has same levels of access, or supposedly so. Should we not have closed systems with what you just said?

A I think there's a bit of a, of a dichotomy that must emerge. There's sort of I think we need to think about the most cutting edge and the most advanced systems. Um, those we will want to ensure are closed for geopolitical reasons, for military reasons, for whatever reasons, like as we develop systems that are, that are genuinely so, so powerful, we will want to keep those closed. That, that doesn't preclude us from, you know, making open Less advanced versions of technology that frankly just have the ability to produce a lot of economic value. Um, and I think that's where we are with llama right now. Like I don't think llama, um, that llama three models are so advanced that you would think of like llama three in and of itself is not a military asset yet. Um, and I think that there's, we're, there's clearly a line underneath which I think it's totally fine to have open models. Um, And I think we, so that's what we need to be thoughtful about is where that line is and when are we getting close to it.

AI assessment note: “the most advanced systems. Um, those we will want to ensure are closed”

Q I respect you, do your best work, and you just have to identify which camp someone's in, and then hopefully, if their skills are there, they should operate to their best. I wish I'd known that when I started, and I didn't, and I just Tried to act out of fear for everyone there. What do you know now that you wish you'd known, and where did you fuck up?

A The biggest one was, um, was actually, you know, in the same era of like, 20, 20, 21, was thinking that hypergrowth as a company meant that you had to hypergrow your team. And so, in those few years, we did what a lot of tech companies did. We, like, doubled, tripled the team year on year, like the, the, you know, from, uh, in 20, 20, we were about a 150 people. By the end of 20, 22, Uh, we were over 700. And it was just this, it was this insane amount of hiring and this incredible amount of, of, of hyper growth as a team. And what I found out is when you hire that quickly, it is impossible to, to do what we've just been talking about, which is maintaining this high bar and maintaining this, this sort of feeling of excellence within the team.

AI assessment note: “thinking that hypergrowth as a company meant that you had to hypergrow your team.”

Q I respect you, do your best work, and you just have to identify which camp someone's in, and then hopefully, if their skills are there, they should operate to their best. I wish I'd known that when I started, and I didn't, and I just Tried to act out of fear for everyone there. What do you know now that you wish you'd known, and where did you fuck up?

A The biggest one was, um, was actually, you know, in the same era of like, 20, 20, 21, was thinking that hypergrowth as a company meant that you had to hypergrow your team. And so, in those few years, we did what a lot of tech companies did. We, like, doubled, tripled the team year on year, like the, the, you know, from, uh, in 20, 20, we were about a 150 people. By the end of 20, 22, Uh, we were over 700. And it was just this, it was this insane amount of hiring and this incredible amount of, of, of hyper growth as a team. And what I found out is when you hire that quickly, it is impossible to, to do what we've just been talking about, which is maintaining this high bar and maintaining this, this sort of feeling of excellence within the team.

AI assessment note: “thinking that hypergrowth as a company meant that you had to hypergrow your team”

Q I thought that is absolute shit. Uh, and I think when you look at like data provisions and what the Chinese government will be willing to do in terms of In terms of data access and data provisions and regulation, I, I think if they are two years behind, that will very quickly catch up. How do you see China being two years behind, and do you agree with that?

A Two years ago, there were probably more than two years behind. So, um, when, you know, when OpenAI first produced GPT-IV in the lab, uh, you know, China were nowhere near that. But just even the past few months, uh, there was a Chinese company, zero one, zero one dot AI, Uh, that produced, uh, a model YI-large, uh, YI-large, that is now the, uh, one of the best models in the world. I think it's just behind, so it's behind GPT-IV-O and, uh, Gemini Claude III Opus, and it's the next model right behind that in the leaderboards. So it's one of the best models in the world. Um, so we've already seen them meaningfully catch up. They are like, you know, Chinese, Chinese LLM and AI capabilities are, I would say right now, pretty close to neck and neck with us capabilities. And I think if you plot the path ahead based on everything we've talked about with data, I mean, I think they're gonna, they have a clear shot at racing forward and racing ahead of us. And I think it comes down to at its core, the, that the CCP system is incredibly good at Um, at taking, uh, very aggressive centralized action and centralized industrial policy to drive forward critical industries. And what we've seen even in the past few years on, or the past few decades, frankly, on solar, um, uh, how China has, or the CCP has been able to make, um, uh, take industrial policy to the point of like being by and large t…

AI assessment note: “right now, pretty close to neck and neck with us capabilities”

Q How's your view changed? Are you most excited now?

A I'm quite excited, um, but I think there's also, there's also reasons to be cautious, and I think that one of the things that happened in the autonomous vehicle craze is that there were a lot of promises that were being made that were divorced from the technical reality, and so a lot of the prominent autonomous vehicle companies, a lot of the prominent organizations were making bolder and bolder promises to be able to raise money Um, and those were, you know, at first they weren't super divorced, but over time they became more and more divorced from the technical realities. And that resulted in this very painful trough where it's sort of the, the promises, uh, weren't met. And so, and so it felt like the entire industry was falling apart. And actually at the end of the day, you know, now we have Waymo's driving around San Francisco. We have Elf, you know, perfectly proper L four autonomous vehicles driving around. Um, Tesla autopilot has gotten really good. And so if we had more measured promises along the way, I think now we would feel amazing about autonomous vehicles. Whereas instead we went through this huge up, this big down, and maybe it's sort of like, uh, on the upswing again. I think this is one of the big concerns I have about generative AI, which is, uh, I hope not, but the same thing might happen again, which is that we have, we have these really big promises that a…

AI assessment note: “I'm quite excited, um, but I think there's also, there's also reasons to be cautious”

Q What would a pro data regulatory stance look like?

A I think there's a few things. I think first there's, um, there's large data sets that are not, um, do not lend proprietary advantages to specific players that need to be sort of centralized and, and made accessible to whole industries. So simple examples, um, safety data. Um, uh, in, in, let's say, aerospace, which is a, which is a hot topic, obviously. Um, but safety data in aerospace should be collectively pooled for the purpose of advancing the entire industry forward. Or the data I mentioned before, or the, the example I mentioned before, fraud, uh, and compliance in financial services should be pooled together and, and should be, you know, uh, should build forward capabilities. Um, so I think there's, like, entire, like, Industrial sectors where there should be some degree of data pooling to just push forward the overall industry. And I think what you need is, um, in, in a lot of consumer facing, uh, areas, you need to, we need to work through a lot of the existing restrictions to make sure that those don't prevent AI progress. So, so one great example here is actually, uh, HIPAA in healthcare. So Um, and, and all the PII and other, other, uh, limitations. Right now, um, uh, you know, HIPAA and PII and all the PII regulations will more or less prevent, uh, patient data from being used to train, um, you know, AI models. But I think we can agree as a, as a, as a civilization…

AI assessment note: “entire Industrial sectors where there should be some degree of data pooling”

Q of per seat pricing. Like he said, you know, Chris had that provocative article, but like everyone talks about the end of per seat pricing. To what extent Do you think we will see the end of per seat pricing in this next wave of software? And especially with the data lens where you could see a more consumption based pricing model aligned. Do you think that truly takes over?

A The reason that per seat pricing, um, uh, doesn't make sense going into the future is that at an enterprise, uh, today, certainly most of the productive work is done by, is done by their employees, done by people. But in a future where you imagine more and more of the work is done by AI agents or AI models, um, then perceived pricing doesn't really make sense, because you want, you know, as a, as a provider of software, provider of, of solutions, you want to make sure that you're capturing the value that you're providing to the people, but also the value that, you know, your agents or your, you know, your AI systems are producing. And so I think that, that shifts a lot of the world towards consumption-based pricing. Um, versus per seat.

AI assessment note: “shifts a lot of the world towards consumption-based pricing. Um, versus per seat.”

Q Given that concern, should we not have closed systems? Obviously, open systems have a lot of benefits, but the challenge with open systems is anyone can use them, and that means that Russia can use them, China can use them, and everyone has same levels of access, or supposedly so. Should we not have closed systems with what you just said?

A I think there's a bit of a, of a dichotomy that must emerge. There's sort of I think we need to think about the most cutting edge and the most advanced systems. Um, those we will want to ensure are closed for geopolitical reasons, for military reasons, for whatever reasons, like as we develop systems that are, that are genuinely so, so powerful, we will want to keep those closed. That, that doesn't preclude us from, you know, making open Less advanced versions of technology that frankly just have the ability to produce a lot of economic value. Um, and I think that's where we are with llama right now. Like I don't think llama, um, that llama three models are so advanced that you would think of like llama three in and of itself is not a military asset yet. Um, and I think that there's, we're, there's clearly a line underneath which I think it's totally fine to have open models. Um, And I think we, so that's what we need to be thoughtful about is where that line is and when are we getting close to it.

AI assessment note: “those we will want to ensure are closed for geopolitical reasons”

Q How's your view changed? Are you most excited now?

A I'm quite excited, um, but I think there's also, there's also reasons to be cautious, and I think that one of the things that happened in the autonomous vehicle craze is that there were a lot of promises that were being made that were divorced from the technical reality, and so a lot of the prominent autonomous vehicle companies, a lot of the prominent organizations were making bolder and bolder promises to be able to raise money Um, and those were, you know, at first they weren't super divorced, but over time they became more and more divorced from the technical realities. And that resulted in this very painful trough where it's sort of the, the promises, uh, weren't met. And so, and so it felt like the entire industry was falling apart. And actually at the end of the day, you know, now we have Waymo's driving around San Francisco. We have Elf, you know, perfectly proper L four autonomous vehicles driving around. Um, Tesla autopilot has gotten really good. And so if we had more measured promises along the way, I think now we would feel amazing about autonomous vehicles. Whereas instead we went through this huge up, this big down, and maybe it's sort of like, uh, on the upswing again. I think this is one of the big concerns I have about generative AI, which is, uh, I hope not, but the same thing might happen again, which is that we have, we have these really big promises that a…

AI assessment note: “I'm quite excited, um, but I think there's also, there's also reasons to be cautious”

Q Why are we not going to get there from internet data? When we think about effective agents, and when we think about software doing the work, not just selling the tools, as I think Sarah Townville said quite well before, why is existing data, ah, not equipped to do that transition from tools to work?

A Yeah. So the, um, the simple answer is like a lot of the, uh, a lot of the thought process and a lot of the thinking that humans go through when they are doing more complex tasks that doesn't get written down on the internet. So for example, if I'm a fraud analyst, um, inside a large bank, you know, I go through, my job is like understanding with, based on a set of transactions that seems suspicious, um, whether or not It's a fraudulent, ah, it's a fraudulent transaction, and I need to analyze all sorts of different pieces of data, and use my deduction, and use all my, like, sort of, ah, human intelligence to make that decision, and that process that I go through, it's not like every, like, I'm writing down step by step, like, oh, I looked at this piece of data, and I looked at this piece of data, and then, based on that, I deduce this, and I'm not writing all that down on the internet to later be crawled, ah, by these models. So, one way to think about is, like, all of the reasoning and thinking That is powering the economy today. None of that gets written down on the internet. And so if you just train on the internet, the model has no ability to learn from all of that.

AI assessment note: “thinking that humans go through when they are doing more complex tasks that doesn't get written down”

Q at the beginning, you know, I spoke to one of the most powerful CTOs in the world the other day, and they said the real breakthrough in kind of this question of are we reaching diminishing returns is like whether we can really solve reasoning. Like, how do you think about our ability to solve reasoning and that impact of data that you mentioned there in helping us navigate that?

A You know, if you look at what these models can do, They're very good at reasoning in situations where they've seen a lot of data before. You know, I think we like to think about these AIs as, uh, as if they're like little human intelligences, but they're, they're very different. Human intelligence and machine intelligence are very different. Humans are very good at sort of, um, humans have a very general form of intelligence. They have this great ability, you know, uh, if a human, if a, if a kid is raised in a very small neighborhood, Um, they can live their whole lives in that small neighborhood, and they can go to an entirely different part of the world, and they can navigate, understand what's going on. No AI system today would be able to do that level of sort of, ah, you know, drag and drop in one situation to another situation and figure out what's going on. So, um, so we have to, I think we have to be cognizant that that's a limitation, but what that means is that, you know, for any situation that we want these models to perform well in, we need to have data Of that situation or that scenario, and actually the model will perform really well. So there's kind of two ways to think about, um, resolving the reasoning gap that exists in these current models. One is, uh, obviously you build some sort of general reasoning capability, which would definitely be a big breakthrough f…

AI assessment note: “there's kind of two ways to think about, um, resolving the reasoning gap”

Q of per seat pricing. Like he said, you know, Chris had that provocative article, but like everyone talks about the end of per seat pricing. To what extent Do you think we will see the end of per seat pricing in this next wave of software? And especially with the data lens where you could see a more consumption based pricing model aligned. Do you think that truly takes over?

A The reason that per seat pricing, um, uh, doesn't make sense going into the future is that at an enterprise, uh, today, certainly most of the productive work is done by, is done by their employees, done by people. But in a future where you imagine more and more of the work is done by AI agents or AI models, um, then perceived pricing doesn't really make sense, because you want, you know, as a, as a provider of software, provider of, of solutions, you want to make sure that you're capturing the value that you're providing to the people, but also the value that, you know, your agents or your, you know, your AI systems are producing. And so I think that, that shifts a lot of the world towards consumption-based pricing. Um, versus per seat.

AI assessment note: “I think that, that shifts a lot of the world towards consumption-based pricing.”

Q What would a pro data regulatory stance look like?

A I think there's a few things. I think first there's, um, there's large data sets that are not, um, do not lend proprietary advantages to specific players that need to be sort of centralized and, and made accessible to whole industries. So simple examples, um, safety data. Um, uh, in, in, let's say, aerospace, which is a, which is a hot topic, obviously. Um, but safety data in aerospace should be collectively pooled for the purpose of advancing the entire industry forward. Or the data I mentioned before, or the, the example I mentioned before, fraud, uh, and compliance in financial services should be pooled together and, and should be, you know, uh, should build forward capabilities. Um, so I think there's, like, entire, like, Industrial sectors where there should be some degree of data pooling to just push forward the overall industry. And I think what you need is, um, in, in a lot of consumer facing, uh, areas, you need to, we need to work through a lot of the existing restrictions to make sure that those don't prevent AI progress. So, so one great example here is actually, uh, HIPAA in healthcare. So Um, and, and all the PII and other, other, uh, limitations. Right now, um, uh, you know, HIPAA and PII and all the PII regulations will more or less prevent, uh, patient data from being used to train, um, you know, AI models. But I think we can agree as a, as a, as a civilization…

AI assessment note: “there should be some degree of data pooling to just push forward the overall industry”

Q 800 people. You are now getting to the, kind of, bigger company size. It is harder, you know, the, kind of, only hire A plus players, or A players. A players, by definition, are rarer. Can you have 800 A players?

A I think the answer is yes, and I think that, um, you know, what we say a lot internally is, how do we hire the Navy SEALs, not, not the Navy, not that there's anything wrong with the Navy, but how do you have, you know, a really small elite group, um, that is really, uh, where you're really hiring the cream of the crop, and this goes down to process, you know, for us, um, at this point in the company, still, I approve every hire, so I will look at The, I will either indirectly interview or look at the interview feedback and look at the, um, understand every single person who we hire to ensure that we're keeping an exceptionally high bar. And that way if there's.

AI assessment note: “I think the answer is yes”

Q How's your view changed? Are you most excited now?

A I'm quite excited, um, but I think there's also, there's also reasons to be cautious, and I think that one of the things that happened in the autonomous vehicle craze is that there were a lot of promises that were being made that were divorced from the technical reality, and so a lot of the prominent autonomous vehicle companies, a lot of the prominent organizations were making bolder and bolder promises to be able to raise money Um, and those were, you know, at first they weren't super divorced, but over time they became more and more divorced from the technical realities. And that resulted in this very painful trough where it's sort of the, the promises, uh, weren't met. And so, and so it felt like the entire industry was falling apart. And actually at the end of the day, you know, now we have Waymo's driving around San Francisco. We have Elf, you know, perfectly proper L four autonomous vehicles driving around. Um, Tesla autopilot has gotten really good. And so if we had more measured promises along the way, I think now we would feel amazing about autonomous vehicles. Whereas instead we went through this huge up, this big down, and maybe it's sort of like, uh, on the upswing again. I think this is one of the big concerns I have about generative AI, which is, uh, I hope not, but the same thing might happen again, which is that we have, we have these really big promises that a…

AI assessment note: “I'm quite excited, um, but I think there's also... reasons to be cautious”

Q Why are we not going to get there from internet data? When we think about effective agents, and when we think about software doing the work, not just selling the tools, as I think Sarah Townville said quite well before, why is existing data, ah, not equipped to do that transition from tools to work?

A Yeah. So the, um, the simple answer is like a lot of the, uh, a lot of the thought process and a lot of the thinking that humans go through when they are doing more complex tasks that doesn't get written down on the internet. So for example, if I'm a fraud analyst, um, inside a large bank, you know, I go through, my job is like understanding with, based on a set of transactions that seems suspicious, um, whether or not It's a fraudulent, ah, it's a fraudulent transaction, and I need to analyze all sorts of different pieces of data, and use my deduction, and use all my, like, sort of, ah, human intelligence to make that decision, and that process that I go through, it's not like every, like, I'm writing down step by step, like, oh, I looked at this piece of data, and I looked at this piece of data, and then, based on that, I deduce this, and I'm not writing all that down on the internet to later be crawled, ah, by these models. So, one way to think about is, like, all of the reasoning and thinking That is powering the economy today. None of that gets written down on the internet. And so if you just train on the internet, the model has no ability to learn from all of that.

AI assessment note: “a lot of the thinking that humans go through... doesn't get written down on the internet.”

Q at the beginning, you know, I spoke to one of the most powerful CTOs in the world the other day, and they said the real breakthrough in kind of this question of are we reaching diminishing returns is like whether we can really solve reasoning. Like, how do you think about our ability to solve reasoning and that impact of data that you mentioned there in helping us navigate that?

A You know, if you look at what these models can do, They're very good at reasoning in situations where they've seen a lot of data before. You know, I think we like to think about these AIs as, uh, as if they're like little human intelligences, but they're, they're very different. Human intelligence and machine intelligence are very different. Humans are very good at sort of, um, humans have a very general form of intelligence. They have this great ability, you know, uh, if a human, if a, if a kid is raised in a very small neighborhood, Um, they can live their whole lives in that small neighborhood, and they can go to an entirely different part of the world, and they can navigate, understand what's going on. No AI system today would be able to do that level of sort of, ah, you know, drag and drop in one situation to another situation and figure out what's going on. So, um, so we have to, I think we have to be cognizant that that's a limitation, but what that means is that, you know, for any situation that we want these models to perform well in, we need to have data Of that situation or that scenario, and actually the model will perform really well. So there's kind of two ways to think about, um, resolving the reasoning gap that exists in these current models. One is, uh, obviously you build some sort of general reasoning capability, which would definitely be a big breakthrough f…

AI assessment note: “two ways to think about, um, resolving the reasoning gap that exists in these current models”

Q of per seat pricing. Like he said, you know, Chris had that provocative article, but like everyone talks about the end of per seat pricing. To what extent Do you think we will see the end of per seat pricing in this next wave of software? And especially with the data lens where you could see a more consumption based pricing model aligned. Do you think that truly takes over?

A The reason that per seat pricing, um, uh, doesn't make sense going into the future is that at an enterprise, uh, today, certainly most of the productive work is done by, is done by their employees, done by people. But in a future where you imagine more and more of the work is done by AI agents or AI models, um, then perceived pricing doesn't really make sense, because you want, you know, as a, as a provider of software, provider of, of solutions, you want to make sure that you're capturing the value that you're providing to the people, but also the value that, you know, your agents or your, you know, your AI systems are producing. And so I think that, that shifts a lot of the world towards consumption-based pricing. Um, versus per seat.

AI assessment note: “I think that, that shifts a lot of the world towards consumption-based pricing.”

Q What would a pro data regulatory stance look like?

A I think there's a few things. I think first there's, um, there's large data sets that are not, um, do not lend proprietary advantages to specific players that need to be sort of centralized and, and made accessible to whole industries. So simple examples, um, safety data. Um, uh, in, in, let's say, aerospace, which is a, which is a hot topic, obviously. Um, but safety data in aerospace should be collectively pooled for the purpose of advancing the entire industry forward. Or the data I mentioned before, or the, the example I mentioned before, fraud, uh, and compliance in financial services should be pooled together and, and should be, you know, uh, should build forward capabilities. Um, so I think there's, like, entire, like, Industrial sectors where there should be some degree of data pooling to just push forward the overall industry. And I think what you need is, um, in, in a lot of consumer facing, uh, areas, you need to, we need to work through a lot of the existing restrictions to make sure that those don't prevent AI progress. So, so one great example here is actually, uh, HIPAA in healthcare. So Um, and, and all the PII and other, other, uh, limitations. Right now, um, uh, you know, HIPAA and PII and all the PII regulations will more or less prevent, uh, patient data from being used to train, um, you know, AI models. But I think we can agree as a, as a, as a civilization…

AI assessment note: “there's like entire like Industrial sectors where there should be some degree of data pooling”

Q Given that concern, should we not have closed systems? Obviously, open systems have a lot of benefits, but the challenge with open systems is anyone can use them, and that means that Russia can use them, China can use them, and everyone has same levels of access, or supposedly so. Should we not have closed systems with what you just said?

A I think there's a bit of a, of a dichotomy that must emerge. There's sort of I think we need to think about the most cutting edge and the most advanced systems. Um, those we will want to ensure are closed for geopolitical reasons, for military reasons, for whatever reasons, like as we develop systems that are, that are genuinely so, so powerful, we will want to keep those closed. That, that doesn't preclude us from, you know, making open Less advanced versions of technology that frankly just have the ability to produce a lot of economic value. Um, and I think that's where we are with llama right now. Like I don't think llama, um, that llama three models are so advanced that you would think of like llama three in and of itself is not a military asset yet. Um, and I think that there's, we're, there's clearly a line underneath which I think it's totally fine to have open models. Um, And I think we, so that's what we need to be thoughtful about is where that line is and when are we getting close to it.

AI assessment note: “the most advanced systems. Um, those we will want to ensure are closed”

Q 800 people. You are now getting to the, kind of, bigger company size. It is harder, you know, the, kind of, only hire A plus players, or A players. A players, by definition, are rarer. Can you have 800 A players?

A I think the answer is yes, and I think that, um, you know, what we say a lot internally is, how do we hire the Navy SEALs, not, not the Navy, not that there's anything wrong with the Navy, but how do you have, you know, a really small elite group, um, that is really, uh, where you're really hiring the cream of the crop, and this goes down to process, you know, for us, um, at this point in the company, still, I approve every hire, so I will look at The, I will either indirectly interview or look at the interview feedback and look at the, um, understand every single person who we hire to ensure that we're keeping an exceptionally high bar. And that way if there's.

AI assessment note: “I think the answer is yes”

Q Speaking of kind of improving the data set that you have there, I am interested because plugging the gaps in terms of the edge cases, I'm seeing more and more synthetic data creation platforms pop up. How do you think about synthetic data creation platforms, and does that change the landscape and the value of data moving forward?

A Yeah, so synthetic data is a very alluring concept. I think that you'll be able to just generate data to match these edge cases. It hasn't really worked in practice in most situations, particularly when it comes to visual data or even textual data. It really hasn't worked, and a lot of the reasons why it's sort of limited in terms of its overall impact is that at the end of the day, whenever you generate data, there's always some sort of weird artifacts or bias in the synthetic data that you generate. You can see this, like, When people generate deepfakes, there's always some weird artifacts in that data. And the thing is, machine learning models are really good at picking up those little points of bias in a way that they, they basically won't generalize very well. And so all that is to say, I think it's, it's this very alluring idea. And if you talk about a high level, it's very exciting. But if you really dig in on like, where is the state of the technology and how impactful is it really? It's nowhere close to actually impact the curve of machine learning yet.

AI assessment note: “It's nowhere close to actually impact the curve of machine learning yet.”

Q I absolutely love that in terms of the drop out to start scale. Can I ask, A, what did your parents say? And B, in terms of risk tolerance and how you assess risk personally, how do you think about your own risk assessment there?

A Yeah, so I had to tell my parents that I was planning on going back to school. Jury's still out if I'm planning to go back to school, but that's what, that's why I had to tell them to let me do it. And then in terms of risk tolerance, I adopt a very risk-seeking perspective, because the reality is that there's not actually that much risk. When you start a company, the worst case, obviously, is that it fails, but even in that case, it's not that bad, especially if you're young, because if you're in that position, you'll still have plenty of opportunity after the company, and so I think a lot of people overweight risk, particularly when it comes to taking a job at a small startup or starting a company, etc., because they think that the negatives will really weigh on them, but I think You see people who start companies and fail, and then start another one, and it succeeds, and they're just going for it, and it's not really that negative of a, of experience.

AI assessment note: “I adopt a very risk-seeking perspective, because the reality is that there's not actually that much risk”

Q I absolutely love that in terms of the drop out to start scale. Can I ask, A, what did your parents say? And B, in terms of risk tolerance and how you assess risk personally, how do you think about your own risk assessment there?

A Yeah, so I had to tell my parents that I was planning on going back to school. Jury's still out if I'm planning to go back to school, but that's what, that's why I had to tell them to let me do it. And then in terms of risk tolerance, I adopt a very risk-seeking perspective, because the reality is that there's not actually that much risk. When you start a company, the worst case, obviously, is that it fails, but even in that case, it's not that bad, especially if you're young, because if you're in that position, you'll still have plenty of opportunity after the company, and so I think a lot of people overweight risk, particularly when it comes to taking a job at a small startup or starting a company, etc., because they think that the negatives will really weigh on them, but I think You see people who start companies and fail, and then start another one, and it succeeds, and they're just going for it, and it's not really that negative of a, of experience.

AI assessment note: “I had to tell my parents that I was planning on going back to school.”

Q Speaking of kind of improving the data set that you have there, I am interested because plugging the gaps in terms of the edge cases, I'm seeing more and more synthetic data creation platforms pop up. How do you think about synthetic data creation platforms, and does that change the landscape and the value of data moving forward?

A Yeah, so synthetic data is a very alluring concept. I think that you'll be able to just generate data to match these edge cases. It hasn't really worked in practice in most situations, particularly when it comes to visual data or even textual data. It really hasn't worked, and a lot of the reasons why it's sort of limited in terms of its overall impact is that at the end of the day, whenever you generate data, there's always some sort of weird artifacts or bias in the synthetic data that you generate. You can see this, like, When people generate deepfakes, there's always some weird artifacts in that data. And the thing is, machine learning models are really good at picking up those little points of bias in a way that they, they basically won't generalize very well. And so all that is to say, I think it's, it's this very alluring idea. And if you talk about a high level, it's very exciting. But if you really dig in on like, where is the state of the technology and how impactful is it really? It's nowhere close to actually impact the curve of machine learning yet.

AI assessment note: “It's nowhere close to actually impact the curve of machine learning yet.”