The Exchanges

Q Do you do that from a legal perspective, or is that something that's a little bit more in the future relative to where code is today?

A Yeah, we're starting to do this now, and actually, like, when I was at DeepMind, a lot of the RL research I did was that, and so when we first got access to GPT-IV, we had the very strong intuition of, okay, you're going to be able to, you know, string a bunch of these model calls or eventually do things like reasoning models where the full agent is differentiable, and Even the first day we got access to GPT-IV, Winston went in his room for 14 hours and just redid a bunch of his associate tasks, and when I looked at the work he was doing, it was essentially like this hacky agentic where he said, okay, I would need to go look up this case law, summarize it, take that summary, use it to draft, and so seeing him do that gave us the intuition very early on of that's the direction this is going, and you can kind of think of associates as Agents. They get this task from a partner that's, hey, I have this high level case strategy. I want to see if I can find a bunch of case law that supports it. Can you go research that? Look it up, cite it, write me a memo. And so a lot of the systems we're starting to build look a lot like that. And I think one interesting direction that the coding labs, the research labs are going is building these RL environments where you deploy these agents and they can interact with a code base and see if they can pass unit tests. And in legal, that RL environm…

AI assessment note: “Yeah, we're starting to do this now”

Q One of the, as you mentioned, like the labs are all very focused on, um, uh, RL scaling in, um, like coding and math domains. Um, I think those is like highly verifiable, right? Not perfectly so, but like how, how do you think about the appropriateness of like law for RL given it's not as easily verifiable?

A Yeah, this, this is one of the biggest problems. And I remember we had conversations early on when we were trying to figure out what is the right Evaluation structure. So I think the hardest thing about legal is most of these tasks are very long form text generation. And so there are definitely subsets of legal work that are super verifiable of go in this data room and just find all the change of control provisions that you can kind of build these traditional data sets. But for something like generate this merger agreement, it's really hard to just give some binary like this is good or this is bad. And I think this has been like a big research problem, like with all the labs we work with, and also internally, there is just this open question of how do you build that reward function? And if you think of what that reward function is at the law firms, it's the partners, right? Like at the end of the day, there is no way to verify this besides the senior partner who's done a bunch of these said, yeah, this looks pretty good. And so internally, these law firms have a bunch of data of here's all the edits that went into this and the feedback. And so We are starting to think about how do you use that to train these reward functions, but I would say that is one of the really big problems, but I think one of the interesting things is, I think you actually have the same problem in progra…

AI assessment note: “there is just this open question of how do you build that reward function”

Q I asked the internet through X, uh, what questions we should ask you. Um, and a popular one was like, why aren't you guys building a law firm? Are you going to build a law firm and compete with all your customers?

A Yeah, no, we, we get this question. And I mean, I think when we first started Harvey and we were doing research, we actually talked to 30 people from Atrium. And I think also, interestingly, Sam and Uh, Jason, who was the GC of OpenAI at the time, and was the GC at Y Combinator when they did the Atrium investment. What struck us was the people who worked there said it was a really good idea, and they were super excited about the prospects, and then there were some challenges around the legal and the execution. But when we dug more into it, the big challenge that they ran into was, you're essentially just building two different companies, right? You're building a law firm, And you're building a tech company, and it's already really hard to, like, build product engineering, do AI, scale sales, and I think the big issue you run into if you try to do both of these is, I think you can only do one thing well, and doing a law firm well is very different than building a software company well. I think that's one point. The bigger point is, for us, it feels like the best outcome is if we can figure out How do we make every law firm? How do we help every law firm become an AI first law firm? Not how do we build one ourselves? And I think the real problem we're trying to solve is can we make every law firm more profitable? And a part of that is how they work with their clients. And can you…

AI assessment note: “you're essentially just building two different companies, right? You're building a law firm, And you're building a tech company”

Q era of AI is that there's deeply technical people building giant companies in really different industries. And you come from a research background. You worked at one of the major labs in terms of foundation models and other areas, RL environments, reinforcement learning. What has been your biggest surprise in terms of transitioning into being a founder and running a company and building something from the ground up like that?

A They may be, maybe not surprised, but biggest like mental model shift is I think the 10 years before Harvey, I was doing a mix of mainly AI research and then trying to start companies, but always largely as an IC. And I think the shift from this started working and scaling just how much I had to change my mental model of the type of company we're building, how you do this at scale, how you operate. I think that was the biggest surprise or like Thing that I've had to, had to change, but it's been a crazy experience kind of going from, you know, Winston and I in an Airbnb to 500 people in like three and a half years. And then I think also how you build these products at scale and kind of the complexity of this industry like that has been like a really hard but interesting experience.

AI assessment note: “biggest like mental model shift is I think the 10 years before Harvey”

Q Exactly. Yeah. Other question was, um, you know, there's a bunch of foresight in starting Harvey when you guys did. When you look forward, do you have a prediction that you think others don't necessarily agree with you right now? That is not mainstream.

A So I have one comment I'll definitely make on the foresight is I think a lot of like, we've gotten comments of like, oh, overnight success and oh, you saw this coming. And I would say, I actually just spent the decade before Harvey trying to start a company like Harvey, so I think it was just, I was super early, and then eventually it was like, oh, now's the right time, and then you were kind of in the right position. I, my guess is, I think people now are catching up to how capability pilled, as you called it, like Winston and I were. I think people in Silicon Valley, I think people have a good sense of where these models are going, but I think generally people Don't appreciate how much better they're going to continue getting.

AI assessment note: “generally people Don't appreciate how much better they're going to continue getting.”

Q Because you're, you're gonna shrink the set of people that are needed to do certain tasks over time, right? Right now, that isn't true. It's augmentation, it's expanding business, but that could happen in the long run. How do you think about the future of law, or what law firms will look like, or, you know, the evolution of all that?

A Yeah, this is a great question. I think it's changed a lot in the past couple years. I think something we are starting to talk with law firms a lot is How do we think of training the future generation of partners where, to your point, these law firms have these leverage ratios where you have a lot of associates but much less partners, and there is value to that because not everyone is going to become a partner, and part of going through that process is how you find, oh, this is the person that I would trust to do this very complex acquisition because they've gone through that experience, and so I think the part I'm optimistic about is if I think about Over 10 years ago when I learned to program, it was super painful, right? Like you had to go on Stack Overflow. It was hard to learn multiple languages because you're like, okay, I'm just going to like learn Python. I'm going to learn TensorFlow. It was just like hard to even learn that. It was hard to ask questions. When I was at Google, you don't want to ask a bunch of questions because people would be like, oh, you don't know that.

AI assessment note: “How do we think of training the future generation of partners where... law firms have these leverage ratios”