Opinion
Weinberg: Standard AI benchmarks are useless for evaluating legal AI
“Most benchmarks are completely useless for us, right? And so we'll get a model, you know, someone will give us early access to a model and they'll say it's way better on all of these benchmarks and we'll respond. It actually isn't like, it's not used as useful…”
Prediction Not checkable as stated
Weinberg: Automated legal tasks will shift to fixed-fee pricing
“I don't think the billable hour is going to just completely disappear. I think what is going to happen is a lot of these tasks that are going to get, you know, AI can automate a lot of that task with a lawyer in the loop. Those tasks will end up being kind of …”
Insight
Weinberg: AI founders must aim to transform entire industries
“If you are using these tools and you don't think that you can take basically AI and apply to X industry and transform the entire industry, I don't think you're thinking ambitiously enough, right?”
Insight
Weinberg: AI platforms must expand modular workflows and collapse UI complexity
“These models can't just one shot all of these really complex legal tasks or in these other domains like tax and provide other professional services. And so what you have to do is you have to build a platform that is kind of constantly expanding and constantly …”
Insight
Weinberg: Enterprise AI requires pairing broad productivity tools with chained vertical workflows
“You need to build productivity tools, and what I mean by this is
Things that are useful for the highest amount of seats, right?
And then you also need to build things that are streamlined vertical workflow from start to finish, right?
And what you can do is…”
Insight
Weinberg: Productivity AI needs lower quality thresholds if work is verifiable
“On the productivity side, the minimum viable quality of that output can be lower because you're selling seats, And at the end of the day, there are multiple people reviewing it, right? And so what you want to do in that state is have just show your work, right…”
Insight
Weinberg: Earning enterprise trust requires targeting the hardest customers first
“We found that no matter what, you eventually have to get the trust of the industry. And the best way to do that is to actually go after the hardest people first.”
Insight
Weinberg: AI drives task displacement rather than job displacement in law
“It is not job. Displacement is task displacement. And I think that's a super important distinction because getting rid of those tasks does not mean the legal industry falls apart. It will evolve.”
Disclosure
Weinberg: Harvey is helping law firms build and sell specialized software
“We were starting to work with firms and basically take, you know, the special things that they do and the way that they practice law and turn it into their specialized system in Harvey. And then they go and sell that to their clients.”
Assertion Not checkable as stated
Weinberg: Law firms perform work at a loss for PE clients
“There is so much legal work that law firms actually do at a loss and they do it at a loss... There's a lot of legal work that they do at a loss in order to get the big deal. So, you know, a lot of law firms will do work for private equity and they will do that…”
Insight
Weinberg: Founders should do every role themselves before hiring for it
“One of the lessons that I definitely want to keep is I do think you should do every single role For a certain amount of time before you hire for it. Almost all of my mishires were because I did not understand what that role was.”
Insight
Weinberg: Spending time with exceptional people is undervalued compared to asking questions
“I think that it is very overvalued how much value you get from asking people that you really respect, very specific questions, I think is overvalued. I think spending time with people that are incredible at whatever they do is undervalued. And I think it's bec…”
Insight
Weinberg: Vertical AI founders should evaluate the 'price per token'
“So one thing that I look at is you can look at this by industry and you can look at it by task, which is how expensive is the token? And I, I've come up with like different ways to call this, but basically if you look at something like that share purchase agre…”
Insight
Weinberg: Intuition for AI progress comes from hammering models through failures
“It seems like there was this large gap between
I log into ChatGPT or I just use GPT-IV from an API or whatever it is, and I try something a couple times versus I'm going to sit there and just hammer on this until it works, right?
and I think if you do that …”
Insight
Weinberg: Vertical AI evaluation requires senior domain experts, not junior staff
“And the reality is you have to hire very good lawyers who can actually evaluate these systems, and the same within tax and these other areas, and they can't be too junior, because if they were too junior and they were able to eval it, they would be senior, rig…”
Insight
Weinberg: Evaluating scoped AI workflows is easier than open-ended copilots
“It's much more scoped, and evaluation is much easier, right? Like, how do you do evaluation on copilot or enterprise GPT, et cetera? That is very difficult. It is much easier to do evaluation if you can do evaluation at each step and make sure that recursively…”
Insight
Weinberg: The fundamental UI for professional services is just email
“And if you actually think about how the UI for professional services is just email, like that's it.”
Prediction Not checkable as stated
Weinberg: AI will allow junior lawyers to do strategic work earlier
“And so what I think will end up happening is the timeline will compress. So you will start being able to actually do the high level strategic work and interact with clients, which is what people really want to do earlier on in your career.”
Insight
Weinberg: Legal AI workflows transfer effectively to tax and finance
“There's so many areas where legal is kind of the tip of the spear and then you can actually just kind of paralyze the same things that you're building and tweak them. And it works really well in other industries.”
Insight
Weinberg: Senior employees protect positions, making critical feedback seekers rare leadership signals
“There are a few folks that will come up to me and basically just not ask for positive feedback and just say, what did I get? Like, what do I need to improve? That is a massive signal because I think that something that happens as you get more senior in your ca…”
Prediction Not checkable as stated
Weinberg: AI will produce specialized ChatGPT-level breakthroughs in coding and medicine
“I think we will end up seeing work completion done in very sophisticated manners, like in medicine and in coding, like a lot in coding and a lot also in, you know, other areas too, that will kind of bring us back to the chat GBT moment, but it will be speciali…”
Winston Weinberg