StarZero All shows Try the podcast clipper →

Aug 24, 2025 · 1h 9m · lennys-podcast

Inside the expert network training every frontier AI model | Garrett Lord

Garrett Lord · 44m spoken Lenny Rachitsky · 18m spoken
0:00 / 0:00
▶ Watch on YouTube →

gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions

Handshake co-founder and CEO Garrett Lord joins Lenny Rachitsky to discuss how Handshake built a $50M+ ARR AI data business by mobilizing its network of academic researchers and PhDs to train frontier AI models. Lord breaks down the technical transition from pre-training to post-training, the operational playbook for executing an internal startup spinout, and the future of work and hiring in an AI-driven economy.

How this conversation actually went

Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 30% of the talking time here. How this is scored →

Lenny as informed peer 3.6 Guest teaching 4.6 Guest disagreement 1.4 Lenny pushing back 1.2
05100:0015:0030:0045:001:00:005:04–9:42 · Lenny as informed peer 4/10 Demystifying AI Training: Pre-Training vs. Post-Training Lenny tries to categorize post-training into RLHF versus fine-tuning. Garrett gently reframes the conceptual framework, explaining that post-training now focuses heavily on reasoning models and trajectory data.9:43–13:03 · Lenny as informed peer 3/10 Handshake's Unique Moat: Academic Expert Networks Garrett details Handshake's academic moat, explaining how PhD networks are required to break frontier models where generalist labelers fail. Lenny acts as an engaged clarifier.13:05–17:50 · Lenny as informed peer 3/10 Inside Expert Data Generation and Flaw Discovery Garrett educates Lenny on how expert data generation works in practice, citing GPQA papers, reasoning step fixes, and subjective domains like educational design.17:51–22:02 · Lenny as informed peer 3/10 Trajectories, Multimodal Data, and Automated Rubrics Lenny asks for technical clarifications on trajectories, data formats, and rubric evaluation. Garrett explains LLM-as-a-judge auto-evals and research turnaround cycles.22:04–27:58 · Lenny as informed peer 4/10 The Future of Human Feedback and Student Employment Lenny challenges Garrett on whether students training AI are essentially coding themselves out of future entry-level employment. Garrett firmly rejects this premise, arguing that AI-native graduates operate like amplified 'Iron Man' workers.28:03–31:48 · Lenny as informed peer 4/10 Market Dynamics, Competitors, and Organic Audience Moats Lenny brings up industry context regarding Kevin Weil and Scale AI. Garrett explains the competitive dynamics and why labs distrust labeling vendors acquired by competitors, highlighting Handshake's zero-CAC advantage.31:49–40:43 · Lenny as informed peer 4/10 Sponsor Segment: Claude by Anthropic Lenny draws an analogy to StackBlitz and Bolt leveraging dormant assets for AI product surges. Garrett walks through Handshake's early network structure and rapid revenue scaling.40:44–45:42 · Lenny as informed peer 4/10 Expert Community Architecture vs. Traditional Labeling Mills Garrett breaks down the CAC/LTV mechanics of traditional labeling mills versus Handshake's university network, sharply contrasting expert treatment with low-cost clickwork farms.45:43–57:12 · Lenny as informed peer 4/10 Operating a Fast-Paced Startup Within a Mature Company Garrett shares tactical management lessons on running a startup within a mature company. Lenny synthesizes key takeaways around founder involvement, team isolation, and rigorous metrics.57:14–1:02:23 · Lenny as informed peer 4/10 Reinventing Job Matching and the Frontier of AI Data Lenny raises the looming industry question of whether frontier models will hit a data wall. Garrett outlines the shift toward specialized scientific tool trajectories and argues synthetic data will not displace human expert input.1:02:26–1:07:38 · Lenny as informed peer 3/10 Founder Mindset and Opportunities in AI A friendly lightning round covering book recommendations, early founder hustle stories like showering in the Princeton pool, and closing recruitment appeals.5:04–9:42 · Guest teaching 6/10 Demystifying AI Training: Pre-Training vs. Post-Training Lenny tries to categorize post-training into RLHF versus fine-tuning. Garrett gently reframes the conceptual framework, explaining that post-training now focuses heavily on reasoning models and trajectory data.9:43–13:03 · Guest teaching 5/10 Handshake's Unique Moat: Academic Expert Networks Garrett details Handshake's academic moat, explaining how PhD networks are required to break frontier models where generalist labelers fail. Lenny acts as an engaged clarifier.13:05–17:50 · Guest teaching 6/10 Inside Expert Data Generation and Flaw Discovery Garrett educates Lenny on how expert data generation works in practice, citing GPQA papers, reasoning step fixes, and subjective domains like educational design.17:51–22:02 · Guest teaching 6/10 Trajectories, Multimodal Data, and Automated Rubrics Lenny asks for technical clarifications on trajectories, data formats, and rubric evaluation. Garrett explains LLM-as-a-judge auto-evals and research turnaround cycles.22:04–27:58 · Guest teaching 4/10 The Future of Human Feedback and Student Employment Lenny challenges Garrett on whether students training AI are essentially coding themselves out of future entry-level employment. Garrett firmly rejects this premise, arguing that AI-native graduates operate like amplified 'Iron Man' workers.28:03–31:48 · Guest teaching 5/10 Market Dynamics, Competitors, and Organic Audience Moats Lenny brings up industry context regarding Kevin Weil and Scale AI. Garrett explains the competitive dynamics and why labs distrust labeling vendors acquired by competitors, highlighting Handshake's zero-CAC advantage.31:49–40:43 · Guest teaching 3/10 Sponsor Segment: Claude by Anthropic Lenny draws an analogy to StackBlitz and Bolt leveraging dormant assets for AI product surges. Garrett walks through Handshake's early network structure and rapid revenue scaling.40:44–45:42 · Guest teaching 5/10 Expert Community Architecture vs. Traditional Labeling Mills Garrett breaks down the CAC/LTV mechanics of traditional labeling mills versus Handshake's university network, sharply contrasting expert treatment with low-cost clickwork farms.45:43–57:12 · Guest teaching 4/10 Operating a Fast-Paced Startup Within a Mature Company Garrett shares tactical management lessons on running a startup within a mature company. Lenny synthesizes key takeaways around founder involvement, team isolation, and rigorous metrics.57:14–1:02:23 · Guest teaching 5/10 Reinventing Job Matching and the Frontier of AI Data Lenny raises the looming industry question of whether frontier models will hit a data wall. Garrett outlines the shift toward specialized scientific tool trajectories and argues synthetic data will not displace human expert input.1:02:26–1:07:38 · Guest teaching 2/10 Founder Mindset and Opportunities in AI A friendly lightning round covering book recommendations, early founder hustle stories like showering in the Princeton pool, and closing recruitment appeals.5:04–9:42 · Guest disagreement 2/10 Demystifying AI Training: Pre-Training vs. Post-Training Lenny tries to categorize post-training into RLHF versus fine-tuning. Garrett gently reframes the conceptual framework, explaining that post-training now focuses heavily on reasoning models and trajectory data.9:43–13:03 · Guest disagreement 1/10 Handshake's Unique Moat: Academic Expert Networks Garrett details Handshake's academic moat, explaining how PhD networks are required to break frontier models where generalist labelers fail. Lenny acts as an engaged clarifier.13:05–17:50 · Guest disagreement 1/10 Inside Expert Data Generation and Flaw Discovery Garrett educates Lenny on how expert data generation works in practice, citing GPQA papers, reasoning step fixes, and subjective domains like educational design.17:51–22:02 · Guest disagreement 1/10 Trajectories, Multimodal Data, and Automated Rubrics Lenny asks for technical clarifications on trajectories, data formats, and rubric evaluation. Garrett explains LLM-as-a-judge auto-evals and research turnaround cycles.22:04–27:58 · Guest disagreement 3/10 The Future of Human Feedback and Student Employment Lenny challenges Garrett on whether students training AI are essentially coding themselves out of future entry-level employment. Garrett firmly rejects this premise, arguing that AI-native graduates operate like amplified 'Iron Man' workers.28:03–31:48 · Guest disagreement 2/10 Market Dynamics, Competitors, and Organic Audience Moats Lenny brings up industry context regarding Kevin Weil and Scale AI. Garrett explains the competitive dynamics and why labs distrust labeling vendors acquired by competitors, highlighting Handshake's zero-CAC advantage.31:49–40:43 · Guest disagreement 1/10 Sponsor Segment: Claude by Anthropic Lenny draws an analogy to StackBlitz and Bolt leveraging dormant assets for AI product surges. Garrett walks through Handshake's early network structure and rapid revenue scaling.40:44–45:42 · Guest disagreement 2/10 Expert Community Architecture vs. Traditional Labeling Mills Garrett breaks down the CAC/LTV mechanics of traditional labeling mills versus Handshake's university network, sharply contrasting expert treatment with low-cost clickwork farms.45:43–57:12 · Guest disagreement 1/10 Operating a Fast-Paced Startup Within a Mature Company Garrett shares tactical management lessons on running a startup within a mature company. Lenny synthesizes key takeaways around founder involvement, team isolation, and rigorous metrics.57:14–1:02:23 · Guest disagreement 2/10 Reinventing Job Matching and the Frontier of AI Data Lenny raises the looming industry question of whether frontier models will hit a data wall. Garrett outlines the shift toward specialized scientific tool trajectories and argues synthetic data will not displace human expert input.1:02:26–1:07:38 · Guest disagreement 0/10 Founder Mindset and Opportunities in AI A friendly lightning round covering book recommendations, early founder hustle stories like showering in the Princeton pool, and closing recruitment appeals.5:04–9:42 · Lenny pushing back 1/10 Demystifying AI Training: Pre-Training vs. Post-Training Lenny tries to categorize post-training into RLHF versus fine-tuning. Garrett gently reframes the conceptual framework, explaining that post-training now focuses heavily on reasoning models and trajectory data.9:43–13:03 · Lenny pushing back 1/10 Handshake's Unique Moat: Academic Expert Networks Garrett details Handshake's academic moat, explaining how PhD networks are required to break frontier models where generalist labelers fail. Lenny acts as an engaged clarifier.13:05–17:50 · Lenny pushing back 1/10 Inside Expert Data Generation and Flaw Discovery Garrett educates Lenny on how expert data generation works in practice, citing GPQA papers, reasoning step fixes, and subjective domains like educational design.17:51–22:02 · Lenny pushing back 1/10 Trajectories, Multimodal Data, and Automated Rubrics Lenny asks for technical clarifications on trajectories, data formats, and rubric evaluation. Garrett explains LLM-as-a-judge auto-evals and research turnaround cycles.22:04–27:58 · Lenny pushing back 3/10 The Future of Human Feedback and Student Employment Lenny challenges Garrett on whether students training AI are essentially coding themselves out of future entry-level employment. Garrett firmly rejects this premise, arguing that AI-native graduates operate like amplified 'Iron Man' workers.28:03–31:48 · Lenny pushing back 1/10 Market Dynamics, Competitors, and Organic Audience Moats Lenny brings up industry context regarding Kevin Weil and Scale AI. Garrett explains the competitive dynamics and why labs distrust labeling vendors acquired by competitors, highlighting Handshake's zero-CAC advantage.31:49–40:43 · Lenny pushing back 1/10 Sponsor Segment: Claude by Anthropic Lenny draws an analogy to StackBlitz and Bolt leveraging dormant assets for AI product surges. Garrett walks through Handshake's early network structure and rapid revenue scaling.40:44–45:42 · Lenny pushing back 1/10 Expert Community Architecture vs. Traditional Labeling Mills Garrett breaks down the CAC/LTV mechanics of traditional labeling mills versus Handshake's university network, sharply contrasting expert treatment with low-cost clickwork farms.45:43–57:12 · Lenny pushing back 1/10 Operating a Fast-Paced Startup Within a Mature Company Garrett shares tactical management lessons on running a startup within a mature company. Lenny synthesizes key takeaways around founder involvement, team isolation, and rigorous metrics.57:14–1:02:23 · Lenny pushing back 2/10 Reinventing Job Matching and the Frontier of AI Data Lenny raises the looming industry question of whether frontier models will hit a data wall. Garrett outlines the shift toward specialized scientific tool trajectories and argues synthetic data will not displace human expert input.1:02:26–1:07:38 · Lenny pushing back 0/10 Founder Mindset and Opportunities in AI A friendly lightning round covering book recommendations, early founder hustle stories like showering in the Princeton pool, and closing recruitment appeals.

speaking balance: gold is Lenny, purple is the guest (3 minute bins)

0:00 · Lenny 76.9% · guest 23.1%0:00 · Lenny 76.9% · guest 23.1%3:00 · Lenny 96.6% · guest 3.4%3:00 · Lenny 96.6% · guest 3.4%6:00 · Lenny 11.7% · guest 88.3%6:00 · Lenny 11.7% · guest 88.3%9:00 · Lenny 5.5% · guest 94.5%9:00 · Lenny 5.5% · guest 94.5%12:00 · Lenny 19.1% · guest 80.9%12:00 · Lenny 19.1% · guest 80.9%15:00 · Lenny 23% · guest 77%15:00 · Lenny 23% · guest 77%18:00 · Lenny 19% · guest 81%18:00 · Lenny 19% · guest 81%21:00 · Lenny 32.9% · guest 67.1%21:00 · Lenny 32.9% · guest 67.1%24:00 · Lenny 23.6% · guest 76.4%24:00 · Lenny 23.6% · guest 76.4%27:00 · Lenny 33.9% · guest 66.1%27:00 · Lenny 33.9% · guest 66.1%30:00 · Lenny 39.6% · guest 60.4%30:00 · Lenny 39.6% · guest 60.4%33:00 · Lenny 22.6% · guest 77.4%33:00 · Lenny 22.6% · guest 77.4%36:00 · Lenny 29.6% · guest 70.4%36:00 · Lenny 29.6% · guest 70.4%39:00 · Lenny 45.1% · guest 54.9%39:00 · Lenny 45.1% · guest 54.9%42:00 · Lenny 0% · guest 100%42:00 · Lenny 0% · guest 100%45:00 · Lenny 32.2% · guest 67.8%45:00 · Lenny 32.2% · guest 67.8%48:00 · Lenny 6.7% · guest 93.3%48:00 · Lenny 6.7% · guest 93.3%51:00 · Lenny 38.4% · guest 61.6%51:00 · Lenny 38.4% · guest 61.6%54:00 · Lenny 0% · guest 100%54:00 · Lenny 0% · guest 100%57:00 · Lenny 16.9% · guest 83.1%57:00 · Lenny 16.9% · guest 83.1%1:00:00 · Lenny 34.8% · guest 65.2%1:00:00 · Lenny 34.8% · guest 65.2%1:03:00 · Lenny 39.4% · guest 60.6%1:03:00 · Lenny 39.4% · guest 60.6%1:06:00 · Lenny 30.2% · guest 69.8%1:06:00 · Lenny 30.2% · guest 69.8%1:09:00 · Lenny 85.7% · guest 14.3%1:09:00 · Lenny 85.7% · guest 14.3%
Sharpest disagreement ▶ 24:42 Garrett rejects the job-displacement narrative

Garrett directly dismisses the premise that student model training hurts early-career job prospects, insisting that real employers see AI-native students as uniquely empowered.

 Hardest push from Lenny ▶ 24:17 Lenny presses on the AI displacement paradox

Lenny directly confronts Garrett with the fundamental irony and economic tension of students training the AI systems that might eliminate their own entry-level opportunities.

 Biggest teaching moment ▶ 8:41 Garrett redefines post-training beyond basic RLHF

Garrett educates Lenny on modern post-training architecture, explaining that frontier labs have moved past simple preference ranking into complex multi-step reasoning and trajectory collection.

 Lenny holds their own ▶ 38:49 Lenny's StackBlitz/Bolt unfair advantage analogy

Lenny demonstrates sharp domain synthesis by comparing Handshake's sudden AI data advantage to StackBlitz's multi-year web OS development paving the way for Bolt's AI boom.
the scores for every segment, with the reasoning behind each
ChapterTopicLenny as informed peerGuest teachingGuest disagreementLenny pushing backWhy
Demystifying AI Training: Pre-Training vs. Post-Training 4621 Lenny tries to categorize post-training into RLHF versus fine-tuning. Garrett gently reframes the conceptual framework, explaining that post-training now focuses heavily on reasoning models and trajectory data.
Handshake's Unique Moat: Academic Expert Networks 3511 Garrett details Handshake's academic moat, explaining how PhD networks are required to break frontier models where generalist labelers fail. Lenny acts as an engaged clarifier.
Inside Expert Data Generation and Flaw Discovery 3611 Garrett educates Lenny on how expert data generation works in practice, citing GPQA papers, reasoning step fixes, and subjective domains like educational design.
Trajectories, Multimodal Data, and Automated Rubrics 3611 Lenny asks for technical clarifications on trajectories, data formats, and rubric evaluation. Garrett explains LLM-as-a-judge auto-evals and research turnaround cycles.
The Future of Human Feedback and Student Employment 4433 Lenny challenges Garrett on whether students training AI are essentially coding themselves out of future entry-level employment. Garrett firmly rejects this premise, arguing that AI-native graduates operate like amplified 'Iron Man' workers.
Market Dynamics, Competitors, and Organic Audience Moats 4521 Lenny brings up industry context regarding Kevin Weil and Scale AI. Garrett explains the competitive dynamics and why labs distrust labeling vendors acquired by competitors, highlighting Handshake's zero-CAC advantage.
Sponsor Segment: Claude by Anthropic 4311 Lenny draws an analogy to StackBlitz and Bolt leveraging dormant assets for AI product surges. Garrett walks through Handshake's early network structure and rapid revenue scaling.
Expert Community Architecture vs. Traditional Labeling Mills 4521 Garrett breaks down the CAC/LTV mechanics of traditional labeling mills versus Handshake's university network, sharply contrasting expert treatment with low-cost clickwork farms.
Operating a Fast-Paced Startup Within a Mature Company 4411 Garrett shares tactical management lessons on running a startup within a mature company. Lenny synthesizes key takeaways around founder involvement, team isolation, and rigorous metrics.
Reinventing Job Matching and the Frontier of AI Data 4522 Lenny raises the looming industry question of whether frontier models will hit a data wall. Garrett outlines the shift toward specialized scientific tool trajectories and argues synthetic data will not displace human expert input.
Founder Mindset and Opportunities in AI 3200 A friendly lightning round covering book recommendations, early founder hustle stories like showering in the Princeton pool, and closing recruitment appeals.

Statements from this episode (18)

Assertion Supported
Lord: AI pre-training gains asymptoted 18 to 24 months ago
“And about 18 months ago, 24 months ago, we started to really see, like, an asymptoting of gains coming from, because they had essentially, like, sucked up all of the knowledge on the internet. And so labs really shifted towards most of the gains now coming fro…”
Garrett Lord Aug 24, 2025 ▶ 6:35
Opinion
Lord: AI models no longer need generalist annotators, only domain experts
“But really what's happened is the models have gotten so good that the generalists are no longer needed. Like what they really need is experts, experts across every area that the models are focused on.”
Garrett Lord Aug 24, 2025 ▶ 11:03
Assertion Not checkable as stated
Frontier AI builders demand real-world tool-use and trajectory data
“Evolving the use cases to also follow what the model builders want, which is they want more. They want more real-world tool use and trajectory-based data as well.”
Garrett Lord Aug 24, 2025 ▶ 15:20
Disclosure
Lord: Handshake engages hundreds of elite music students for AI multimodal data
“And we're engaging like thousands or not thousands, like probably hundreds of top music students at, you know, the weed music schools in the country who are improving models, understanding of music.”
Garrett Lord Aug 24, 2025 ▶ 18:27
Prediction Not checkable as stated
Lord: Humans will be needed to train AI for the next decade
“And I think for as long as models are improving, humans will be needed in this process. And when you talk to the lead scientists and researchers at these labs, it's like the data types will evolve and what they're trying to capture and collect, but you know, t…”
Garrett Lord Aug 24, 2025 ▶ 23:20
Assertion Supported
Lord: 100% of the Fortune 500 uses Handshake
“There's like a million companies that use Handshake. Like we have 101 hundred percent of the Fortune 500 uses Handshake.”
Garrett Lord Aug 24, 2025 ▶ 24:56
Assertion Partly supported
Handshake pays academic AI data contributors $100 to $200 per hour
“And like they can make like a hundred, a 152 hundred dollars an hour in their area and their field of expertise.”
Garrett Lord Aug 24, 2025 ▶ 27:22
Insight
Lord: The only moat in human AI data is audience access
“We like to say like the only moat in human data is access to an audience.”
Garrett Lord Aug 24, 2025 ▶ 30:35
Assertion Not checkable as stated
Lord: Handshake has 500,000 PhDs and 3M master's students
“We have, you know, more PhDs. 500,000 of them use Handshake than any other platform. We have three million master's students who are, you know, in-school or alumni.”
Garrett Lord Aug 24, 2025 ▶ 34:09
Assertion Not checkable as stated
Lord: Handshake AI works with seven frontier AI labs
“Fast forward to today, we're working with Seven of the frontier labs, basically every lab that's doing work and building the best large language models.”
Garrett Lord Aug 24, 2025 ▶ 36:06
Prediction Not checkable as stated
Lord: Handshake AI will exceed $100M revenue in year one
“I think we'll blow through that number, but yeah.”
Garrett Lord Aug 24, 2025 ▶ 36:43
Assertion Not checkable as stated
A leading AI data competitor spends tens of millions monthly on ads
“One of the leading players has like, they have like 200 recruiters. It's like unsustainable. There are like 200 people on LinkedIn sending individual messages to acquire these people because there's no brand, there's no trust. They spend, you know, they're spe…”
Garrett Lord Aug 24, 2025 ▶ 41:19
Disclosure
Handshake pre-builds AI training data and licenses it to multiple labs
“There's certain swim lanes where we're actually pre-building data and selling that data to all the labs. So we can do this thing where You know, we produce one unit of data ourselves. We pay for it, almost like a movie production. We pay for a unit of data. An…”
Garrett Lord Aug 24, 2025 ▶ 42:56
Assertion Supported
Handshake powers 92% of top 500 US schools with zero CAC
“We have no customer acquisition cost, because we partner with 1600 universities, power 92% of the top 500 schools in the country. We power almost every institution and community college in the country. We have no customer acquisition cost to acquire the people…”
Garrett Lord Aug 24, 2025 ▶ 44:05
Assertion Partly supported
Lord: Handshake's core business has reached $200M ARR
“Our core business is a two hundred million dollars error.”
Garrett Lord Aug 24, 2025 ▶ 47:14
Insight
Frontier AI labs have practically unlimited demand for high-quality data
“In this market, there's essentially like unlimited demand. Like if you can produce high quality volumes of data you most likely will be able to sell whatever you produce.”
Garrett Lord Aug 24, 2025 ▶ 48:00
Prediction Not checkable as stated
Lord: Hiring managers will not manually review resumes in five years
“The hiring manager process of like reviewing 200 resumes. Are you kidding me? I'm going to sit there and review 200 resumes. Like not a chance five years from now, right? Like students manually making cover, like not a chance, right?”
Garrett Lord Aug 24, 2025 ▶ 58:44
Prediction Not checkable as stated
Lord: Synthetic data will not dominate frontier AI training
“Synthetic data has a role to play and like in verifiable domains, but like what we consistently hear from companies is like, you know, their synthetic data is not going to dominate.”
Garrett Lord Aug 24, 2025 ▶ 1:02:01
Made with StarZero

Turn any episode into a week of clips.

This entire site, over 300 episodes transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Try the podcast clipper Learn more