The Exchanges

Q What is the end-to-end task you're most excited about that you think Harvey will be able to do this year?

A Yeah, filing an S-IV, um, I think is something that is, the reason I like that process is it is a combination of external data, internal data, and a million steps, right? Um, and I think of workflows as basically a bunch of agentic systems that need to combine together, right? And if you think about knowledge work, professional services, legal, honestly, any swath of that, What you are doing is manipulating things based off of your internal context, external context, whether that's external data, whether that's data from your, whoever your customer is, your client is, et cetera, and then process of this is how, you know, this is how you do an LBO. This is how you do side letter compliance. And then there's another step of that, which is this is how you do side letter compliance for this particular private equity firm, right? This is market for this particular clause. And the more complicated the workflow is, the more you have to combine all of those different elements together.

AI assessment note: “filing an S-IV, um, I think is something that is”

Q And expectation of quality from, let's say, like, an associate or a model is very high. Like, how do you, how did you go about building that trust because the, like, you didn't start at the capabilities you have today?

A Yeah, I'll start from one kind of, like, high-level distinction between two pieces. So if you go back to the productivity versus, like, specific specialized output, right? On the productivity side, the minimum viable quality of that output can be lower because you're selling seats, And at the end of the day, there are multiple people reviewing it, right? Um, and so what you want to do in that state is have just show your work, right? Like that is the most important thing. You want to mimic exactly how a senior associate reviews the work of a junior associate. So you say, this is why I did this. This is the information that I pulled. And here's an inline citation to the literal sentence that I pulled it from. Is this correct or not? Right.

AI assessment note: “what you want to do in that state is have just show your work”

Q The legal industry is one of reasoning. How much has the growth of models and interest in developing models that, um, scale test time inference impacted you guys?

A Massively in a good way. Um, the best way to think about this is if you are building a system where you are trying to basically break down every single problem into a sub problem because the models can't quite do it, that just unlocks different pieces of it. So let me give you an example of this. We go back to that antitrust example, right? Where the first step of it is we're trying to take all of the, you know, target financials, your acquirer's financials, and just say in all these different countries, Yeah, this is what you need to file. Well, the next step to that would be, can you help it actually do all of the filings, right? And we were having trouble figuring out how to do that. Now with reasoning models, you can start unlocking those steps, right? And so the best way to think about this is we are constantly building all of the steps that we can and being on the cutting edge. And then when a model improves, that just pushes our ability to go out the next cutting edge more. Um, and then the other thing too, is just cost going down is incredible for us, right? We try, you know, we're not optimizing for cost at all times right now. We're optimizing for quality. And if the prices go down, that makes it so that we can increase our quality across every single user base much faster.

AI assessment note: “Massively in a good way. Um, the best way to think about this”

Q Uh, I'm sure that's a learning experience. You feel like you got a focus on agency, right? And, you know, hiring and the business, like, what did you get wrong?

A Yeah, I mean, I don't think we have, like, 12 hours to do this. Um, a lot, a lot of things. I think if I had to say there's one thing that I got wrong over everything else, it is figuring out when to scale yourself. Um, I have a certain kind of, like, tendency for how I work, and there are some things I, I want to keep. So one of the lessons that I definitely want to keep is I do think you should do every single role For a certain amount of time before you hire for it. Almost all of my mishires were because I did not understand what that role was. And maybe it's a lack of my experience. I don't know, but there's a hands-on piece. Having said that you can't scale a company by wanting to be hands-on in everything at all times. Right. Um, and I think that I didn't spend enough time transitioning from, I mean, the beginning of last year, we were 40 people transitioning from everyone knew what was going on. Everyone had all the context because They were working with me directly. Right.

AI assessment note: “one thing that I got wrong over everything else, it is figuring out when to scale yourself”

Q right around there, California ran a massive surplus of almost a hundred billion dollars. And, um, that was just three, four years ago. And it's really unclear to me what actually happened to all that money. And now we're running a big deficit. Um, where do you think California more broadly should be investing in its own future? And what are the big things that should go fix right now?

A It, a lot of it goes back to infrastructure. I was on the governor's committee, the COVID committee and whatnot. I was amazed to learn how many kids did not have access to the internet. This is the fifth largest economy in the world, and we had kids that had to go home for school, but couldn't connect on the internet. How could that be? Um, we've got demands on our power grid that are exceeding what we can supply. We need to go fix that. We certainly have a vegetation overgrowth problem with forest fires. We need to go fix that. The water delivery problem. We need to go fix that. We need to upgrade our schools, our school system. Just, I think we get back to basic things of saying, what would make residents of Los Angeles lives better and more livable and safer and give them an opportunity to prosper? And let's organize our budget dollars around that, and we have to start prioritizing, but to your point, how did we blow so much money? I, I really don't know. I've never heard a good explanation. I don't know if you've got one.

AI assessment note: “a lot of it goes back to infrastructure... power grid... vegetation overgrowth... water delivery”

Q that anybody can access, right? The, the GPT for us of the world or GPT for the cloud sonnets, et cetera. Um, and then there's all the stuff you've built on top of it to actually make this work well for your specific use case and for, um, customer support agents. Cause you, could you tell us a bit more about what you all have had to build over time?

A Of course. Uh, like you said, everyone has the same access to the same models. We see ourselves very much as a software company, and we're obviously doing a lot of work around AI and like using AI models a lot, but I would argue that most, you know, applications nowadays are, they're real software companies and AI models are kind of tools that everyone can use. And so most of the sort of alpha or most of the specials stuff that you build is on top of models. It's either the orchestration layer or the software around it. Uh, for us, there's been a big focus on both. Uh, the orchestration layer is kind of how you can use all these different models together. You probably have evals set up that measure how good each model is at certain things. You put them together and the whole goal of putting them together is to mold it around the business logic of the customer. That's part one. The other thing you build is just very classic software, right? You have this AI agent there. It's all the things I was saying before, like, you know, Transparency is a big piece. It's, you really don't want this to feel like a black box. That's just their answering questions. And so how can you build all the tooling to see like, okay, what's the data that the agent's using? How's, what steps is it taking? Um, can I analyze all these conversations that are coming in? If you have a million conversations, i…

AI assessment note: “It's either the orchestration layer or the software around it. Uh, for us,”

Q advice. Maybe they don't invest in each other. Like, there's kind of a thriving community, and every five to seven years, it kind of shifts who it is, and I feel like, you know, the IOI sort of math Olympiad community or coding competition communities are kind of very engaged right now. Is there any formal version of that, or are you all just kind of informally helping each other?

A Uh, yeah. I mean, I angel invested in a lot of the companies you just listed. A lot of their founders are angel investors in our company. It's, it's very informal. Obviously it's just like, you know, casual friends helping each other. I think the main thing is that, uh, with company building, there's just a lot of, a lot of service area, right? As you know, it's just like, how do you hire people? How do you like do sales? Like, how do you build this thing? Like, how do you structure like comp or like, I don't know, there's infinite things. So yeah, having the other data points is obviously super helpful. So I hang out with them quite often, play games, play card games. It's a Chinese version of bridge that I play with a lot of these folks quite often. And, uh, it's just, it's fun where you just kind of hang out. Everyone's kind of in this relative same stage of life. And so, yeah, like you said, there is definitely a lot of camaraderie and help that goes around.

AI assessment note: “It's, it's very informal. Obviously it's just like, you know, casual friends”

Q Yeah. I mean, what you describe is a pretty broad platform based approach. Um, I think there's a handful of companies who are sort of in your general space or market. How do you feel that modal differentiates from them?

A I think, first of all, we, we're cloud native. Like, we're just like cloud maximalists. Like, we went all in and said, like, wait a second, we're going to build a multi-tenant platform that runs everyone's computer. And the benefits of that are, like, very tremendous, because, like, we could just do capacity management much better, and that's one of the ways we can offer, like, instantaneous access to hundreds of GPUs if you need to. Like, you can do these, like, very bursty things, and we just give you lots of GPUs, right? I, I think the other benefit, or the other sort of differentiation is to be very general purpose. Uh, we focus on sort of what I think, as I mentioned, like high code, like we run custom code in our containers, in our infrastructure, which is a harder problem. Like containerization and running user code in a safe way is a hard problem. And then dealing with container cold start. And like I mentioned, we have to build our own scheduler. We have to build our own container runtime in our own file system to boot containers very quickly. Uh, and, and I think so unlike many other vendors, like they're only focused on say inference or maybe only LMs. Uh, our, our approach has always been to build a very general purpose platform and, and, and sort of, you know, in the long run, I, I hope to sort of that, that, that sort of manifestation will be more clear because I …

AI assessment note: “the other sort of differentiation is to be very general purpose.”

Q Um, I, if I remember correctly, you were, you were an IOI gold medalist. Yeah, that's right. And obviously you think a lot about code and coding, and how do you think that changes with AI over time? Or do you have any contrarian predictions on, on what happens there?

A I don't know if this is contrarian, but like, I actually think that like, you know, this is just like one out of many improvements in developer productivity. And, you know, you look back at like, you know, whatever, like compilers was originally like, you know, a tool that made developers more productive and then like higher level programming languages and databases and cloud and all these things. And so like, I actually don't know if like AI is like, you know, different than any of those changes in the hindsight. And so, and, and, and by the way, like every time that's happened, you know, It turns out like there's so much latent demand for software that actually like the number of software engineers goes up. So like, I feel like you look back at like, you know, last like four years of software development, like every decade engineers get like 10 times more productive due to better frameworks or better, you know, tooling or whatever. And, and it turns out actually that just unlocks more latent demand for software engineers. So I, I'm very bullish on software engineers. I think it would take a lot to sort of destroy that demand. I think people look at a lot of like AI as like a kind of fix something, but, but in my opinion, it's like, No, it's just gonna unlock more latent demand for more things. So I'm very bullish on software engineering.

AI assessment note: “this is just like one out of many improvements in developer productivity”

Q centers, right? So, you know, if you imagine These chips in servers and racks of servers and big rows of them in the data center. Like we have cooled large data centers for a long time with fans. Like how does this fit into the, I guess, alternative set of fans and liquid cooling and like how, how has, uh, um, large scale AI training changed the game at all?

A So this, uh, this technology that we have, this materials level technology using synthetic diamond, it actually fits with any other cooling technology that's, that's used today. Uh, today, uh, the cooling techniques that are used are really at the data center level, uh, where they do, uh, airflow containment and air containment, uh, you know, keeping the air from mixing. It's, uh, at the rack level where you were using, uh, liquid cooling, you know, CDUs and manifold and pumping liquids, uh, right to the devices to, to keep them cool, uh, or using fans. Uh, and you have it at the, uh, at the chip and package level where, uh, Uh, people are using techniques to, uh, either speed up the chips, right, uh, make more, uh, more transactions on the chip so that you get more efficiency, uh, transactions per watt, uh, or in packaging, uh, doing things like fan-out packaging where you're spreading out the heat, uh, as much as possible for any, uh, any chip or group of chips. Also things like, uh, uh, like HVM, uh, memory you can stack up right in the same package with the chip. And, uh, and give it more efficiency and essentially also more transactional robots. So these are all techniques that are being used today. And the beauty of our approach is that it works with all of these. Uh, you can, you can use, uh, diamond cooling by itself, or you can match it with anything else that you're d…

AI assessment note: “this materials level technology using synthetic diamond, it actually fits with any other cooling technology”

Q to, um, run locally without handling that developer environment mess yourself. There's this concept of like a GPT wrapper company. Right. And so I, I think there were a number of companies that were less generously, like some system prompts and like a well SEO website. Uh, you guys open sourced your system prompts. So, uh, and like a lot of the code for Bolt. Can you explain that strategically?

A As we were building Bolt, um, yeah, I mean, over the past couple of years, there's been a ton of more simple sort of wrappers that have come out, you know, around these, uh, frontier models. And, and the problem is like whenever the next model comes or whenever, you know, one of the AI labs eventually integrates that into their chat products or whatever, uh, those companies, you know, kind of tend to, tend to go away pretty quickly. So when we were working on Bolt, I mean, one of the big advantages we have is we've been building this web container technology for five years, and it's like pretty, pretty difficult stuff to do. Um, and so when we were going to build, uh, to actually launch Bolt, when we were building the system prompts and kind of the user interface around it, When we looked out there, there's not a lot of other folks doing this where, where they actually could open source their system prompts and kind of open source their products. So you could see how it's actually made. And for us, uh, you know, we just, we felt that it was inevitable that someone would, uh, you know, get our AI model to dump out our system prompts anyways, but also we, we, we felt there's kind of something missing in the open source world where, where we see, like, we come from a web developer platform. Right. Open source is key for innovation for everyone. And to date, a lot of these companie…

AI assessment note: “one of the big advantages we have is we've been building this web container technology”

Q And in terms of like your North Star of how you organize, um, the team and invest, uh, you obviously come from a research background yourself. Like how much do you think, um, you know, cohere success is dependent on core models versus other, you know, platform and go to market support investments you make?

A It's all of the above. Like the models are the foundation. And if you're building on a foundation that, um, Doesn't meet the customer's needs, then there's no hope. And so the models are crucial and, um, it's like the heart of the company, but in the enterprise world, things like customer support, reliability, security, these are all key. And so we've heavily invested on both sides. We're not just a modeling organization. We're a modeling and go to market organization. Um, And increasingly, product is becoming a priority for Cohere, and so figuring out ways to shorten time to value for our customers. Um, yeah, over the past, like, 18 months, Since the enterprise world sort of woke up to the technology, we've watched, we've watched folks build with our models, seeing what they're trying to accomplish, seeing the common mistakes that they make. That's been helpful. It's been sometimes frustrating, right? Watching the same mistake again and again. But we think there's a huge opportunity to be able to help enterprises avoid those mistakes and implement things right the first time. And so that's really where we're pushing towards.

AI assessment note: “It's all of the above. Like the models are the foundation.”

Q Do you think in like a traditional hype cycle for enterprise technologies, probably for most technologies, but in particular enterprise, um, uh, you know, there's this trough of disillusionment concept of people get very excited about something and ends up being harder to apply or more expensive than they thought. Do we see that in AI?

A I'm sure we see some of it for sure. Um, But I think honestly, like the core technology is still improving at a steady clip and new applications are getting unlocked every few months. So I, I don't think we're in that trough of disillusionment yet. Yeah. It feels like we're super early. It feels like we're really, really early. And if you look at the market, this technology just unlocks an entire new set of things that you can build. You just fundamentally couldn't build them before, and now you can. And so there's a resurfacing of technology, products, systems that's underway. Even if we didn't train a single new language model, like, okay, all the data centers blow up. We can't improve the LLM. We only have what we have today. There's a half decade of work to go integrate this into the economy, to build all these things, to build the, you know, uh, RFP, insurance RFP response Bought to build the healthcare record summarizer. Like there's a half decade of just resurfacing to go do. So there's a lot of work ahead of us. I think we're kind of past that point. There was a question of, oh, is there too much hype? Is this technology actually going to be useful? But it's in the hands of a hundred million people now, hundreds of millions of people now. It's in production. There's very clear value. The project is now Putting it to work and delivering it, uh, to the world.

AI assessment note: “I don't think we're in that trough of disillusionment yet.”

Q enterprise customer will invest in pre-training is, I think, a bit more controversial. I believe some of the lab leaders would say, like, nobody should be touching this, and it doesn't make any sense for people from a scale of compute and data, data curation effort required, and just sort of the talent required to do pre-training in any sort of competitive way. Like, how would you react to that?

A I think if you're building like a, if you're a big enterprise and you're sitting on a ton of data, like hundreds of billions of tokens of data, um, pre-training is a real lever that you're able to pull. I think for most like SMBs and certainly startups, it makes no sense that you should not be pre-training a model. Um, but if you're a large enterprise, I think it's, it should be a serious consideration. The question is how much pre-training? It's not like you have to start from scratch and do a, you know, fifty million dollar training run, but you can do a fraction, you could do a five million dollar training run. That's what we've seen succeed. These sort of continuation pre-training efforts. Um, so yeah, that, that's one of the offerings that we have, but of course we don't jump straight into that. You don't need to Spend massively if you don't want to, and usually Uh, the enterprise buying cycle or, or technology adoption cycle is quite slow. And so you have time to move back into it. I would say it's totally at the customer's discretion. Um, but to the folks who say that no one should be pre-training.

AI assessment note: “pre-training is a real lever that you're able to pull”

Q Maybe if we go to, um, projection and we'll hit on a few things that you've mentioned as well, um, where are we in scaling laws? Like how much capability improvement do you expect over the next few years?

A We're, we're pretty far along, I would say. Like we're starting to enter into a sort of flat part of the curve, um, and we're certainly past the point where if you just interact with a model, You can know how smart it is. Like the, the vibe checks, they're losing utility. And so instead, what you need to do is you need to get experts to measure within very specific domains like physics, math, uh, chemistry, biology, um, You need to get experts to actually assess the quality of these models because the average person can't tell the difference at this stage between generations. Yes, like there's still much more to go do, uh, but those gains are going to be felt in very specialized areas and have impacts on more researchy, um, more researchy domains. I think for enterprises and the general sorts of tasks that they want to automate or tools that they want to build, The technology is already good enough or close enough that a, a little bit of customization will get them there.

AI assessment note: “we're starting to enter into a sort of flat part of the curve”

Q What do you think is the driver in that difference of opinion?

A I don't know. I, I think maybe I'm A little bit more in the weeds of the practical frustrations of the technology, where it breaks, where it's slow, where it, we start to see things plateau or slow down, um, And perhaps others are more, maybe they're more optimistic. Maybe, maybe they see, um, they see a curve increasing and they just think it goes on forever. Like that will just continue arbitrarily, which I, I disagree with. I think there's, there's friction points. Like there is genuinely friction that enters in. Like maybe even if in theory, you know, like a neural net is a universal approximator, it can learn anything to universally approximate. You would need to build a neural net the size of the universe. So like there's some fundamental barriers to reaching limits that people extrapolate out to that I think will, um, bound the practically realizable, um, forms of this technology.

AI assessment note: “I think maybe I'm A little bit more in the weeds of the practical frustrations”

Q Maybe this is a good time to, to ask you to describe for our listeners, most of whom are technical or somewhere in the tech field, but also a broader business audience. Like, you know, how, how does Alphaproof work overall architecturally?

A Uh, yes, sure. So, um, AlphaProof is based on this thing called AlphaZero. So maybe let me start there. AlphaZero has this program we developed to, you know, like be basically kind of solve kind of perfect information board games. Um, and, um, and the way it works is, is, is also based on a reinforcement learning algorithm. So you can think of AlphaZero as maybe three components, like, uh, one, a neural network, uh, two, kind of, uh, large scale reinforcement learning, uh, basically learning from trial and, uh, trial and errors. And, and three, it also has like this planning and kind of search component to it, to try to, given the current situation, trying to search for kind of the best answer. Um, and, uh, it turned out that, you know, like maybe we weren't very imagining that when we were doing chess, but if you can handle kind of infinite action spaces, uh, then, you know, instead of kind of looking for a chess move, you can look for, for instance, a line of a proof. Um, and so that's what we, we tried to do when we, we, we started off a proof. It's basically, can we look, can we, our action space is to generate lines of proofs, uh, and, and maybe very importantly, we used a formal language to do that. So basically, uh, another way to say that is we use kind of code to write maths, uh, and it's become quite popular recently. Um, and, and the, the advantage of that is that on…

AI assessment note: “AlphaProof is based on this thing called AlphaZero... our action space is to generate lines of proofs”

Q NVIDIA has moved into larger and larger, let's say, like, unit of support for customers. I think about it going from single chip to, you know, server to rack, and VL-CNII. How do you think about that progression? Like, what, what's next? Like, should NVIDIA do full data center?

A In fact, we built full data centers. The way that we build everything, unless you're building If you're developing software, you need the computer in its full manifestation. Um, we don't, we don't build PowerPoint slides and ship the chips, and we build a whole data center. And until we get the whole data center built up, how do you know the software works? Until you get the whole data center built up, how do you know your, you know, Your fabric works and all the things that you expected the efficiencies to be, how do you know it's going to really work at the scale? And, and that's the reason why, that's the reason why it's not unusual to see somebody's actual performance be dramatically lower than their peak performance as shown in PowerPoint slides. And, and, and it's, computing is just not used to, it's not what it used to be. You know, I say that the new unit of computing is the data center. That's to us.

AI assessment note: “In fact, we built full data centers.”

Q Five and a bit. Okay. Um, it's a totally insane thing to try as a startup to, um, you know, go up against the CFTC. Can you talk about that?

A We hold two core beliefs. I think one, prediction markets Have the potential to be the next large thing in financial markets. I, I do genuinely think that, like, if you think about it, uh, from first principles, like, people like to take exposure on things. They like to speculate. They, sometimes they want to hedge. They like, or, you know, it could be either betting or hedging, but, but it feels like the current instruments are a bit like, you know, you, you have things like, okay, you have stocks, that's fine, but you also have, like, options and futures. They're all just so loaded. It's like, so Wall Street-y and, like, people don't really care. Um, But if you think about like prediction markets, what they do is like they, they achieve a similar, similar sort of like urge or they, they, I think they're targeting the similar market, but instead of like these kind of traditional boring financial instruments, you're, you're taking positions on things you care about. Like I have a view on the weather. I have a view on climate. I have a view on AI, which a lot of people have a view on now. And then you can basically take a stake in that. So I think that one, they're going to be very big. I think so. And then two, I have yet to see a financial market that has gone really big. Without being properly regulated. I have yet to see that in history. It's, it's always like you have to do…

AI assessment note: “we spent actually, you know, close to three years getting regulated upfront by the CFTC”

Q How do you think about retail versus like large market makers and institutional monsters?

A I think of it as more of falling on the spectrum that goes from like lots of trades, small size to less frequent trading, but big size. So I think the way that the market evolves is like, because we have lower liquidity now, we target retail and then we scale our liquidity as we scale with retail. And then we scale a type of customer over time. You build an initial layer of liquidity. And we already, I think, have that at CalShe, where like now we have This community, this very dedicated community of people that forecast these events, they price them, and then as they price these different events, you know, like, will it rain tomorrow? Is climate change getting worse or better? Is COVID gonna come back? As you price them, like, the mere fact of getting a price, you can ramp up liquidity. You can get more and more liquidity over time, and as you get more liquidity, you attract the large institutions that may, if you're a big institution, you want to hedge against cyber risk, or a new bill that may pass in Congress that you hate, or other. So it's, I think, a progression. That's how we think about it. Elections specifically, we're seeing a lot of institutional flow. It just said, like, we're, we're a bit late in the game, but the institutions are, take time, compliance take time to onboard, but we're seeing a lot of institutional flow. There is demand for, like, ten million dolla…

AI assessment note: “I think of it as more of falling on the spectrum that goes from”

Q Uh, maybe like just thinking about, you know, the, the coming election, uh, how should people think about prediction markets versus polling?

A There's a lot of confusion around this. Polling is asking people who they're going to vote for, right? Like, so, so assume you can go to, you can survey every single person in Pennsylvania, and then ask them, you know, um, who you're gonna vote for, and then you count it all up, and then you get that Trump is 51% of people, and Kamala is 49% of people, then you, it's guaranteed that Trump is gonna win Pennsylvania, right? Like, that's a guarantee. Now, obviously polls don't work like that. Polls are biased. A lot of them are biased. You should not trust most of them. There's a few that you can trust, but assuming the unbiased ones, like, they don't survey everyone in Pennsylvania, but they survey a good sample, hopefully well done, and then they tell you, we think that it's going to be 5149. The 5149, that one percent difference of polling that Trump has over Kamala, um, that's definitive, right? That's Trump winning. That's like, you know, if the poll is reasonably accurate, that, that, you know, Prediction markets are not doing the same thing, right? Prediction markets are just pricing the odds of Trump versus Kamala winning, right? They are for, like, they're just giving a probability. We think there's a, you know, today right now, it's a 55% chance that Trump is winning. That does not mean that Trump is up five percent on average over Kamala on the polls nationally. That, t…

AI assessment note: “Prediction markets are not doing the same thing, right? Prediction markets are just pricing the odds”

Q for example, um, I think crews ran into some issues in San Francisco where there was activists like putting cones on the cars and trying to stop them and doing other things. And so it seems smart to, to start in Arizona. I was just sort of curious, what are the criteria that led you to, to start that as a sort of a test bed or a place to?

A So I guess, you know, it depends on the different time horizon. So, you know, in the fourth, on the fourth generation, we picked, uh, a deployment area that was kind of medium complexity, uh, and the goal there was, I mentioned, is to kind of go end to end. So we picked an environment where we thought it was, you know, we check enough of the boxes to, you know, help us learn the most important things that we wanted to learn and de-risk, right? While, and that was the deployment. Uh, and then there's the development of the system. So for the development of the system, you want to go after the hardest problems possible, right? You want to go after the densest environments. You want to go after the harshest weather. So we've kind of in parallel been doing that. So we've made a decision to deploy, you know, in Chandler in, you know, to learn from the end to end system while, you know, pushing on developing the system. Then when we, we've, uh, you know, learned enough and we made that discontinuous jump to the fifth generation of our driver. And then we said, okay, like that's the platform we believe that we want to take to scale.

AI assessment note: “we picked a deployment area that was kind of medium complexity”

Q forward a year or two, like now, you know, Crack the nut. Waymo is looking at scale and, um, probably more about the business. Like, do you think of robo taxis at scale as the, the near term business plan? Are there other modalities or like deployment? Like, if I think about this as just like a CapEx problem, like other avenues that are important for you guys to explore.

A That's the main one. Right hailing is the main one that we're focusing on. So we're, you know, focused on technology. We're focused on the product. We are learning from our users. We're every day, we're earning trust. Uh, and we're setting up the ecosystem of, you know, partnerships in that space. So that's our primary focus, right? We are very excited about, uh, the commercial opportunity there. Uh, it's not the only one. I always, you know, think of Waymo as, you know, a technology company. We're building a generalizable Waymo driver, right? With the mission to build the world's most trusted driver. And we want to deploy that driver, not just in ride hailing, right? There's, as I said, there's more than three trillion miles in the US. There's more than 10, you know, trillion miles driven, you know, worldwide. Uh, so the vision and the mission is to deploy the Waymo driver in, you know, different commercial products and different applications and maybe different, you know, modalities across all of that spectrum. So that includes, uh, you know, things like, uh, deliveries, things like, you know, long haul trucking. It includes, uh, you know, things like personally owned vehicles. Uh, but right now we're very focused on retailing.

AI assessment note: “That's the main one. Right hailing is the main one that we're focusing on.”

Q that there was companies like Zoox that Amazon bought, where They kind of hollowed out the inside of the car because you no longer needed the steering column and everything else, and they put seats facing each other, almost like a London cab. Uh, do you have any thoughts on what that experience will look like in the future as more and more things move to autonomous self-driving ride-hailing systems?

A Yeah, so designing a car, uh, around the passengers makes total sense to me. Like we, you know, in the past, we've designed cars around, you know, primarily the driver. Right. Uh, if, you know, it's the Waymo driver, it's all about the, the rider experience. Right. So we have, uh, done, you know, quite a bit of work on the sixth generation of the, you know, the Waymo driver and the car, and the car is designed, uh, around with the passenger in mind. Right. So it is more spacious. Uh, it is all about the user experience. You know, you have, you know, flat floors, you have, you know, lower, uh, floor for entry. You have doors that slide to the side. So it's all about, you know, getting in. So absolutely, you know, there's different aspects of it. Like, you know, we don't have cars facing each other. I think there's, you know, it's an open question. Like some people get, uh, you know, nauseous when you do that. Like you kind of want to, you know, there's benefits on facing forward. But, you know, all of that, I think will be for us as an industry to figure out as soon as we move forward. But I think the, the key, the key point, it becomes, you know, much more like the design is around the rider, not around the driver.

AI assessment note: “we don't have cars facing each other. I think there's, you know, it's an open question.”

Q really surprising. Like people really pushed on, we want this to exist for a long time. We want to be able to pay for it. Um, and so there was that kind of really interesting market pull. Why, why do you think there was so much interest or need for this or demand for it? Or, you know, what does Braintrust do and how does that really impact your customers?

A You know, many of our customers had actually built, uh, early customers had built like internal versions of Braintrust Before we engaged with them. And, uh, there's a couple of things that sort of came out of that. One is it helped them gain an appreciation for how hard the problem is. Evals sound really easy. Oh, it's just a for loop, you know, and then I look at, I console.log the, you know, for loop as I go and I look at the results. Um, but the reality is like, uh, you know, the faster you can eval, the faster you can look at eval results, which start to get really complicated as you start doing things with agents and so on. Um, the faster you can actually iterate and build stuff. It is actually a pretty hard problem to, to do evals well. And many of our early customers, um, who were kind of like the pioneers in, in AI engineering, um, had learned that the hard way. Um, and I think the other problem is that, uh, you know, folks, especially folks, you know, like Brian, for example, they saw that AI would be a pervasive technology throughout the whole org. Not just a project that, you know, Brian might babysit and, and work on with one team. And, um, having a really consistent and, um, standardized, you know, way of doing things was really important. I remember early on, um, Brian pointed me to the Vercel docs, and he said, one of the things I love about this is that, um, whe…

AI assessment note: “it helped them gain an appreciation for how hard the problem is”

Q you spend a lot of your career is on sort of databases and data infrastructure and things like that. So you had the BP engineering at single store, which I think, um, was renowned for really having an exceptional, like, um, database centric team. How do you think about the data infrastructure that exists for the AI world today? What's, what's needed? What's lacking? What works well? What doesn't work?

A The, the, the shift is that people have hoarded lots and lots of semi-useful data in data warehouses. Prior to LLMs, uh, there was actually this whole industry around AI where, you know, companies like DataRobot, for example, would come in and help you train models based on these Proprietary structured data that you've collected in your super proprietary data warehouse. And I think the, the, the big insight or the crazy, you know, non-intuitive thing about, um, LLMs is that something trained on the internet outperforms, uh, what an enterprise can produce with their own data trained, um, on data in a data warehouse. And I think Not only, um, is the nature of, like, the data processing problem different, but the value of data is actually, uh, you know, and how we think about the value of data is very, very different. Like, just hoarding data about your, you know, claims history or transaction history, it might not actually be that useful. Um, the real question is, like, how do you, you know, uh, construct a model which is really good at reasoning about the problems that you're working on, and I think the way that enterprises will Um, collect data and leverage it into, you know, these AI processes does not look like doing ETL on a data warehouse that's, you know, running in, in Amazon or something like that. I think, um, it's gonna totally change. And, and I've seen, um, you know,…

AI assessment note: “does not look like doing ETL on a data warehouse that's, you know, running in, in Amazon”

Q Do you think that was just like a different problem set in terms of traditional ML and the applications of it are different from what JNI can do, or do you think it was something else?

A Well, I went through this myself, uh, watching the technology that we built to do document extraction at Impura become, you know, totally irrelevant. Um, and, uh, Personally, I, I think it's an emotional thing. Like, you, you try GPT-III for the first time, and first of all, you know, back then at least, um, it was kind of snarky, and so that was a little bit irritating, um, and it, it was also just way better at everything than anything you could possibly train, and I think that is so fundamentally disruptive to, you know, a lot of companies, a lot of people's individual identity, Um, it, it just is not easy to wrap your head around if, uh, you've been doing AI and ML for a while. So I, I think it was largely an emotional thing. You could argue that there's a cost, security, privacy, whatever element of it, but the companies that were sort of on the leading edge, they were able to figure that out pretty quickly. Um, you know, now I think more companies have come along the journey, and I've seen a lot of really smart ML and data science people embrace LLMs and bring a lot of the sort of rigor That is still relevant around evals and measurement and, you know, um, prototyping and so on and become these like AI platform teams. Usually it's a combination of people with product engineering backgrounds and, um, you know, a few folks with statistics or data science backgrounds. Um, an…

AI assessment note: “Personally, I, I think it's an emotional thing.”

Q Have you seen any other, um, shifts in terms of, uh, usage of specific languages or, um, tooling or other things that's happened with this wave of AI?

A Yeah, I think the, the biggest thing I've seen over the past six months is, People dropping the use of frameworks. So early on, I think people thought that AI is this, you know, really unique thing. And just like, you know, Ruby on Rails or whatever, um, we're gonna need to build new kinds of applications, uh, with new kinds of frameworks to be able to, to build AI software. And, uh, really, I think people have walked back from that and they now think of AI as kind of like a core part of their software engineering, um, as a whole. And so, AI is now kind of, like, pervasively spreading throughout people's code base, um, and it's not constrained to what you can create with, you know, a single framework.

AI assessment note: “the biggest thing I've seen over the past six months is, People dropping the use of frameworks.”

Q While you were at Yale, you published this very influential essay, um, about Amazon and the antitrust paradox. Um, can you describe the theory behind the essay and, you know, how that might differ from traditional antitrust and if it still applies in your, your thinking today?

A The big caveat here is that, uh, I wrote this paper as a kind of a law student. It's very different from being a law enforcer. Um, but that paper Followed a lot of this market research that I had been doing, um, and before I went to law school, I spent a good six months just talking to a lot of the businesses that were selling through Amazon, and then also talking to various investors and market analysts about how they viewed Amazon from a more long-term perspective. Uh, this was a time when, you know, Amazon's returns were still fairly low, and there was just a question about what the end game here is gonna be. And it was really interesting through talking to those set of market participants, they were already seeing and experiencing the structural dominance that this company was starting to achieve, even though that structural dominance was not necessarily resulting in higher consumer prices right away. And so the SA was a way to use Amazon as a way to tell a broader story about the history of our antitrust laws and how we had evolved from, uh, you know, Viewing this issue of, of dominance and, and monopolization through a broader prism, uh, and a broader understanding of the different ways that companies can exercise their monopoly power to a much narrower, thinned out version of just viewing short-term price effects and output effects primarily as the metric of whether you …

AI assessment note: “use Amazon as a way to tell a broader story about the history of our antitrust laws”

Q is positive for the consumer in some cases, and the trade-off of taking away part of the positive distribution of M&A outcomes is worth that. That's, that's kind of the trade-off I'm hearing. But from the entrepreneur and investor perspective, there is, ah, there feels like there's an inability to predict, you know, when that enforcement is going to happen. What advice would you have for people from that ecosystem?

A I mean, the big piece of advice is, you know, if you're selling to the company that's already the monopolist in a particular market, that is more likely to raise issues than if it's, you know, selling to a company that doesn't really have a presence in the market. The agency has also put out new merger guidelines at the tail end of last year that lay out kind of what are the main frameworks that we use to assess whether a deal may reduce competition or not. I also, you know, think it's worth noting that there may be Kind of a tension between what's good in the short term versus what's good in the long term, right? Partly how we've gotten to the markets of today where you have, you know, five major players for the most part is because they were allowed to buy up some of these nascent competitors and, You know, lead to a market that was much more consolidated and much less competitive for startups. I imagine that that means that the set of potential buyers and the set of potential suitors is much smaller. Right. And even if you want to sell a market in which you have, you know, eight or nine or 10 potential buyers is going to be better for the startup than a market where you just have one or two. Both in terms of your negotiating leverage, you know, the valuation you might get, your ability to negotiate better terms. And so, you know, the market that we want in the longterm is on…

AI assessment note: “if you're selling to the company that's already the monopolist in a particular market”