Apr 11, 2025 · 1h 5m · 20product
20Product: How Scale AI and Harvey Build Product | Why PMs Are Wrong: They are not the CEOs of the Product | How to do Pre and Post Mortems Effectively and How to Nail PRDs | The Future of Product Management in a World of AI with Aatish Nayak
0:00 / 0:00
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Aatish Nayak, Head of Product at Harvey and former product leader at Scale AI, shares practical frameworks on building hypergrowth AI products, rethinking the product management role, and designing collaborative AI user experiences. He also details real-world LLM evaluation methodologies, developer tooling trends, and why human domain experts remain essential in AI deployment.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Harry holds 27.4% of the talking time here. How this is scored →
Harry as informed peer 3.8 Guest teaching 4.3 Guest disagreement 2.2 Harry pushing back 2.8
speaking balance: gold is Harry, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 22:52 Disagreement on dictatorial management
Aatish rejects Harry's assertion that pure dictatorships are best for speed, insisting on benevolent dictatorships that explain context to avoid thrash.
Hardest push from Harry ▶ 22:52 Harry demands dictatorial product authorityHarry explicitly rejects collaborative discussion in product management, claiming talk is cheap and asserting a preference for dictatorships.
Biggest teaching moment ▶ 43:50 Claude 3.7 legal reasoning evaluationAatish educates Harry on why public legal benchmarks fail, explaining multi-clause rubrics and disclosing that Claude 3.7 outperforms OpenAI on long-form legal reasoning.
Harry holds his own ▶ 45:10 OpenAI consumer revenue counter-factHarry demonstrates deep market knowledge by countering Aatish's claim about model labs, pointing out OpenAI's $12.9 billion consumer product revenue.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Harry as informed peer | Guest teaching | Guest disagreement | Harry pushing back | Why |
|---|---|---|---|---|---|---|
| Transitioning from Engineering to Product Management | 3 | 3 | 1 | 1 | Harry asks Aatish about his transition from software engineering to product management. Harry contributes Scott Galloway's philosophy on focusing on what you are good at rather than what you love, which Aatish agrees with. | |
| Scaling Scale AI: Lessons from Frontier Customers | 4 | 4 | 2 | 4 | Aatish explains how Scale AI built products by leaning heavily into frontier customers. Harry pushes back with standard startup wisdom against building custom features, prompting Aatish to reframe how to separate custom requests from broad market signals. | |
| Reframing the PM Role: WD-40, Not Glue | 5 | 5 | 3 | 3 | Aatish reframes the PM role from being the 'glue' or 'CEO of the product' to being 'WD-40' that lubricates customer-engineering interaction. Harry brings in relevant quotes from Shopify leadership to validate the critique of main character syndrome. | |
| The Power of Markets and Scale AI's Vertical Shifts | 5 | 4 | 2 | 4 | Harry brings a venture capitalist lens to market selection and jokes about European versus American pricing discipline. He questions whether data labeling was intrinsically a good market, and Aatish explains Scale AI's strategic pivots across different verticals. | |
| Distribution vs. Product and the Evolution of Chat UX | 4 | 5 | 3 | 4 | Harry questions whether product moats can endure in AI given rapid commoditization. Aatish argues that chat is merely the 'command line MS-DOS' stage of AI interfaces and explains how the Ikea effect builds product retention. | |
| Managing Operational Breakdowns in Hypergrowth | 6 | 5 | 4 | 6 | Harry forcefully asserts his belief in dictatorial leadership over discussion for speed. Aatish counters by advocating for 'benevolent dictatorships' that share context to avoid thrash, sharing candid mistakes made at Harvey. | |
| Strategic Memos, Frameworks, and Rapid Prototyping | 3 | 4 | 2 | 2 | Aatish details Harvey's internal strategic memo hierarchy and why prototyping should precede PRDs. Harry asks clarifying questions on the necessity of design cycles in an era of rapid AI prototyping. | |
| Structuring PRDs and Avoiding Feature Factories | 2 | 5 | 1 | 1 | Aatish outlines what constitutes a strong PRD and warns against 'feature factories'. Harry humorously asks for a step-by-step breakdown of PRD review meetings from an outsider's perspective. | |
| Balancing Technical Debt and Feature Development | 4 | 4 | 2 | 2 | Harry cites insights from Microsoft's CTO regarding AI's ability to eradicate technical debt. Aatish explains how product teams must tie technical debt cleanup to customer revenue outcomes and time-box refactoring. | |
| Premortems and Concentric Circle User Testing | 3 | 4 | 1 | 2 | Aatish differentiates retrospectives, postmortems, and premortems. Harry quotes Mike Tyson regarding unexpected operational punches, and Aatish explains concentric circle user testing. | |
| Lessons from User Testing Pitfalls in Harvey Vault | 4 | 5 | 2 | 3 | Aatish gives transparent examples of product mistakes made with Harvey Vault and Assistant modes. Harry criticizes model dropdown selectors in AI software, which Aatish agrees was a trap built for Silicon Valley power users. | |
| Legal AI Evals & Evaluating Claude 3.7 vs. OpenAI | 4 | 6 | 2 | 3 | Harry cites Clean's CPO regarding divergence between public benchmarks and internal evals. Aatish breaks down Big Law Bench and reveals that Claude 3.7 outperforms OpenAI models on long-form legal reasoning. | |
| Model Landscape Evolution and Developer Workflows | 5 | 4 | 3 | 5 | Harry challenges Aatish's view on model labs needing to become product companies by highlighting OpenAI's $12.9 billion consumer revenue. Aatish discusses developer tool moats across Cursor, Codium, and Replit. | |
| Domain Expertise in Product & Cultural Barriers to AGI | 4 | 5 | 3 | 3 | Aatish asserts that domain expertise will dictate the future of product and claims human culture is a bottleneck to AGI. Harry expresses humorous disbelief at Valley founders using AI models for personal emotional therapy. | |
| Human Context in AI Agent Deployment | 3 | 4 | 2 | 2 | Aatish argues agents need human context and trust to deliver complex enterprise work. Harry compares low-level legal tasks like rental agreements to high-stakes legal work. | |
| Quickfire: Embracing Chaos for Personal Growth | 3 | 3 | 2 | 1 | Harry kicks off the quickfire by sharing his own controversial political views. Aatish pulls up his personal notes app to share his belief that true fulfillment requires embracing chaos and conflict. | |
| Quickfire: Career Guidance for Emerging Professionals | 2 | 3 | 1 | 1 | Aatish shares career advice for graduates entering AI startups and discloses that only about 20% of Harvey's codebase is currently written by AI. | |
| Quickfire: SF Talent Dynamics & Outstanding Product Strategies | 5 | 4 | 3 | 4 | Harry argues London is superior for AI startups due to lower employee churn. Aatish pushes back, claiming the density of experienced executives in San Francisco is irreplaceable. |
Statements from this episode (36)
Nayak: PMs Suffer From Main Character Syndrome and Try to Be CEO
“PMs have a main character syndrome sometimes that they have to be the CEO of the product, be this face of the product.”
Nayak: Domain Experts Are Increasingly Driving Primary Product Decisions
“Domain experts are driving more of the product decisions.”
Nayak: Claude 3.7 Outperforms Other Models in Harvey's Legal Reasoning Evals
“We did these evals recently. Claude three seven, in particular, for legal reasoning, it's better at long form legal reasoning and drafting long form outputs.”
Stebbings: Scale AI Will Reach $2 Billion in Revenue by Late 2025
“Just raised it at twenty-five billion dollars. It ends this year at two billion dollars.”
Nayak: AI Product Teams Should Over-Fit to Frontier Customers
“Find the customers who are really good at what they do, and then, you know, chase them, even over a fit to them as long as you have conviction that, you know, everyone else will follow.”
Nayak: Scale AI Partnered With OpenAI on Early GPT-2 RLHF
“Scale partnered with open AI very, very early on before chat GPT came out. This was like very early on RLHF when they were trying to tune models to summarize better based off of Reddit passages. And this is on GPT two.”
Nayak: Scale AI Flew Engineers Globally to Observe Labeling Tool Usage
“And so we would actually even fly engineers out to training centers all around the world to literally observe how. People are using the labeling products and, you know, different scale products, and prototype and build right there.”
Nayak: Hypergrowth PMs Should Act as WD-40, Not Organizational Glue
“The framing that you should probably have is you're not glue, you're WD-forty.”
Stebbings: American Venture Capital Investors Have Zero Discipline on Price
“The Americans have zero discipline on price, respectfully.”
Nayak: Scale AI labeled e-commerce data for Meta, Instacart, and DoorDash
“Like I started the e-commerce team at scale where we were labeling e-commerce data, like from Meta, from Instacart, from DoorDash.”
Nayak: Foundational AI models are not products in themselves
“I don't think the foundational models are products. You're seeing this with OpenAI right now. They're more pivoting to a product company, and Satya even said this.”
Nayak: Text-Based AI Chat Is Merely the MS-DOS Command-Line Phase
“Definitely not. It's the command line starting point of this new frontier, like the MS-DOS was, you know, way back when.”
Nayak: Hypergrowth Means Scaling Revenue and Headcount 1.5x Every Quarter
“So maybe defining hypergrowth is every three to six months, let's say, your revenue is more than 1.5 x, 1.2 x, your Employee count is more than 1.5 to X. Like it's a different company every single quarter, let's say, or, you know, half.”
Stebbings: Dictatorial Leadership Is Essential for Startup Speed and Efficiency
“Now, I disagree with that. I believe in, bluntly, dictatorships. I think they're much more efficient. And they, when in a world where speed is everything, they allow for a much more efficient and speedy process.”
Nayak: Early dictatorial mandates caused internal organizational thrash at Harvey
“So I think particularly early on at Harvey, we did the dictatorship. Me and Winston are just like, we're going to do X, Y, and Z and Go, go, go. I think again, as you hyper grow, you assume everyone has the same context that you do. And that's just not the cas…”
Nayak: Harvey Uses a 'Land, Expand, Lock In' Product Strategy Framework
“One framework that we have at Harvey is land, expand, lock in. What things can we build slash do to help us land new customers? What can we do to help us expand and grow usage? And then third is how do we make ourselves a moat and very sticky with customers? T…”
Nayak: Harvey PMs Build Product Prototypes Using Cursor and Claude
“PMs on my team create prototypes all the time using cursor or using a Claude to show, you know, show their ideas.”
Nayak: AI Product Teams Should Always Prototype Before Writing PRDs
“I actually always say prototype before the PRD, whenever you have an idea, whether it's engineers, designers, PMs, whatever, you should do the prototype before, because you're not going to really know how to build the product or what the exact UX should be, pa…”
Nayak: Technical debt cleanup requires evidence of revenue or customer impact
“You actually want evidence of technical debt affecting revenue, affecting customer outcomes. And you want to compile the evidence because otherwise if you don't have evidence or you just refer it to as like technical debt overall, it is just like, you can go f…”
Nayak: Harvey Tests New Features With Internal Lawyers Before External Rollouts
“The first thing we do when we build something is. We have an extensive set of lawyers internally doing various roles. We give it to those lawyers and they test it and they give feedback and we iterate. And then we have selected design partners that are consist…”
Nayak: Legal AI Users Prefer Direct Grid Control Over Prompt Auto-Decomposition
“The thing that we did wrong there was actually people want to just make those terms in the grid itself. People don't want the AI to convert it. They just want to say for these 10,000 agreements, I just want this one thing created in the grid itself. They want …”
Nayak: AI products should route queries automatically, not force model choices
“That is an example of where the right user experience is the AI should just pick for you. The AI should just know what user query and intent is, or it should ask you questions to understand it better. And then it should just route you to the best system. For t…”
Nayak: OpenAI is an early investor in Harvey and primary model provider
“Our models of choice are open AI right now... So they've been investors in us from the beginning. We've gotten early access to open AI Models for a long time. We've built some custom solutions with them. For the most part, that's still the case.”
Stebbings: OpenAI Is Now a $12.9 Billion Revenue Consumer Product Company
“OpenAI, which is deliberately chosen, it is now going to be a consumer product company, 12.9 billion in revenue consumer product company, I think very clearly.”
Nayak: Cloud Providers Will Drive AI Inference Margins Down to Zero
“The cloud companies, they're going to run the best software. They're going to, you know, drive margins to the ground. And so competing on inference with cloud companies is really hard over time. And so if all your revenue comes from inference and, you know, de…”
Nayak: Uses Claude and Replit Agent Mode for Rapid Prototyping
“I currently use actually Claude for prototyping because I actually don't want to deal too much with the very nuanced stuff that Windsor for Cursor has. And so I ended up using Claude and then I've actually been using also Replit agent mode. The reason being is…”
Nayak: Harvey's Engineering Team Currently Prefers Cursor Over Competing IDEs
“They prefer Cursor, I believe is the latest. I think it's mostly because we haven't procured Windsurf yet.”
Nayak: Claude Is Empathetic for Personal Advice, Whereas ChatGPT Is Not
“Claude does. Jatchputi doesn't. It's not as empathetic.”
Nayak: Consumers Trust Humans Using AI, Not Standalone AI Agents
“Ultimately, I think the humans don't just always trust AI. They trust other humans using AI. I think, again, depending on the domain, the right thing is more likely that the agent partners with the producer of the work to deliver for the consumer.”
Nayak: Engineers Should Retain Coding Skills Rather Than Rushing Into PM Roles
“The, one of the most valuable things you have is your ability to code if you're an engineer, let's say, and I wish I stuck to it more. But I think it's super important to stick to that at least because it's much better with vibe coding or whatever now.”
Nayak: Approximately 20% of Harvey's Codebase Is Written by AI
“I mean, I think it's probably, it's not that much. It's probably like, 20%.”
Nayak: Software unit testing should be entirely automated by AI
“Individual engineers still use AI somewhat for writing their own unit tests, but that whole layer of unit testing should just be all AI.”
Nayak: Harvey focuses on legal productivity tools over end-to-end automation
“Right now we have to focus on Harvey, the productivity suite, the software that lawyers use, and not the Harvey that, that does the work end to end.”
Stebbings: Building AI Companies Is Easier in London Than San Francisco
“I say that it's easier to build an AI company in London than it is San Francisco. Everyone goes, wah, wah, wah, wah, wah. And it's like, we have the supply of AI talent from some of the best institutions and universities, but crucially, we don't have the churn…”
Nayak: Anthropic Is Surpassing OpenAI in Attracting Top AI Talent
“I would have said OpenAI maybe six months ago, but I think Anthropic is really starting to appeal to a lot more people.”
Nayak: Perplexity AI's Latency-Focused User Experience Is Second to None
“I do think perplexity is honestly just killing it. Like I have a lot of respect for that team and the speed that they're moving. Like, Sure. You can say, oh, they're doing a lot of things now, but I think the focus on the core experience of making an answer re…”