Sep 18, 2025 · 1h 7m · lennys-podcast
Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this interview, Mercor CEO Brendan Foody joins Lenny Rachitsky to discuss how the emergence of expert-driven AI evaluations propelled Mercor's historic scaling from $1 million to $500 million in revenue run rate. Foody explores the mechanics of reinforcement learning environments, dispels near-term superintelligence alarmism, and outlines actionable frameworks for founders and professionals navigating the AI economy.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 32.2% of the talking time here. How this is scored →
Lenny as informed peer 3.4 Guest teaching 3.7 Guest disagreement 0.6 Lenny pushing back 0.9
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 17:30 Dismissing superintelligence job displacement timelines
Brendan counters conventional AI alarmist narratives about imminent superintelligence and job obsolescence, arguing that models remain incapable of basic tasks like drafting emails or scheduling calendars.
Hardest push from Lenny ▶ 22:15 Challenging the definition of elastic job demandLenny presses Brendan to clarify what he means by elasticity in the workforce, challenging whether it refers to generalist skillsets or specific high-demand industry capacities.
Biggest teaching moment ▶ 15:20 Educating on the mechanics of RLAIF vs RLHFBrendan systematically breaks down how human-written rubrics replace slow RLHF human rankings, allowing automated reinforcement learning from AI feedback to scale model training.
Lenny holds their own ▶ 1:01:56 Lenny demos his custom voice-assistant hardware buildLenny demonstrates his technical product experimentation by showing off his custom-wired 'Parrot GPT' hardware project mounted inside a stuffed owl.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| The Era of Evals: Models as Products and PRDs | 3 | 5 | 1 | 1 | Lenny opens by referencing Brendan's pinned tweet and Sarah Guo's quote about evals, positioning evals as a confusing topic for many. Brendan clarifies the concept with an intuitive product framing, explaining that evals act as PRDs and sales collateral for foundation models. | |
| Mercor's Journey: From Bootstrapped Startup to Human Data Frontier | 4 | 4 | 0 | 1 | Lenny frames the rapid ascent of AI data startups and categorizes the landscape into foundational models, vibe-coding apps, and data curation companies. Brendan details Mercor's transition from international generalist staffing to high-end expert sourcing for top AI labs. | |
| How Domain Experts Write Evals and Enable RLAIF | 4 | 6 | 1 | 2 | Lenny asks concrete questions about what domain experts actually do day-to-day and self-identifies as the layperson asking for the audience. Brendan educates Lenny on why the industry is shifting from supervised fine-tuning and RLHF toward RLAIF using rubric-based verifiers. | |
| The Future of Work and the RL Environment Economy | 3 | 5 | 2 | 1 | Lenny asks whether human evaluators will eventually become obsolete, citing a tweet about humans existing solely to generate RL data. Brendan rejects the near-term displacement narrative, asserting that humans will build RL environments for decades as models struggle with basic tool use and long-horizon tasks. | |
| Navigating AI Careers: Elastic Demand and Tool Fluency | 4 | 4 | 1 | 2 | Lenny probes into what students and young professionals should study, asking Brendan to specify which jobs remain elastic. Brendan contrasts low-elasticity fields like accounting with high-elasticity domains like software engineering where higher productivity spurs greater aggregate demand. | |
| Reimagining Global Labor Markets with AI-Powered Matching | 4 | 3 | 1 | 1 | Lenny shares his own ongoing research on how AI has flooded job applications and necessitated automated filtering on the recruiter side. Brendan agrees, explaining why Mercor views itself fundamentally as a labor marketplace rather than a generic data vendor. | |
| Sponsor: Enterpret Customer Intelligence and Voice of Customer | 3 | 5 | 1 | 1 | Following the sponsor read, Lenny relays an anecdote about medical x-ray analysis in ChatGPT to ask whether experts train pre- or post-training data. Brendan educates Lenny on how pre-training ingests broad tokens while post-training experts provide reasoning rubrics and rewards. | |
| Talent Curation: Power Laws, Creative Domains, and Fast Turnaround | 3 | 4 | 0 | 1 | Lenny inquires about compensation rates, project turnaround times, and whether creative writing expertise is valued alongside hard technical domains. Brendan shares metrics, including their $95/hr median pay and hiring comedy writers from the Harvard Lampoon to improve humor in models. | |
| Hypergrowth Drivers: Finding Market Pull and True Product-Market Fit | 3 | 3 | 0 | 0 | Lenny asks how Mercor uncovered hypergrowth demand before raising institutional venture funding. Brendan recounts pitching the founding xAI team while in college and observing incumbents neglect talent quality and payment reliability. | |
| Mercor's Core Values: Can-Do Attitude, High Standards, and Intensity | 3 | 3 | 1 | 1 | Lenny brings up the controversial '996' startup work culture debate, inviting Brendan to explain Mercor's intensity and high standards. Brendan clarifies that Mercor avoids rigid hourly mandates, focusing instead on mission alignment and hiring top tier talent. | |
| Early Entrepreneurship: Donut Dynasty and the Power of Initiative | 2 | 2 | 0 | 0 | Lenny prompts Brendan to share stories from his earlier entrepreneurial projects to extract lessons on founder initiative. Brendan entertains Lenny with the story of running 'Donut Dynasty' in middle school and dodging school restrictions. | |
| Debunking Near-Term Superintelligence and Envisioning AI Abundance | 4 | 4 | 1 | 1 | Lenny references David Sacks' commentary and questions whether model capabilities are plateauing short of superintelligence. Brendan agrees that 3-year AGI predictions are unrealistic, arguing that genuine capability expansion will depend on rigorous, multi-year post-training evals. | |
| AI Corner: Daily Workflows, Thought Partners, and Hardware Experiments | 5 | 2 | 0 | 0 | In AI Corner, Brendan explains how he uses ChatGPT Voice mode as a thought partner, prompting Lenny to showcase a custom wearable hardware project ('Parrot GPT') built into a stuffed owl. Both exchange enthusiastic notes on voice interfaces. | |
| Lightning Round: Dyslexia, Focusing on Strengths, and Media Favorites | 2 | 2 | 0 | 0 | Lenny wraps up with lightning round questions and invites Brendan to discuss managing dyslexia as a high-growth startup CEO. Brendan describes reframing dyslexia as an asset that forces reliance on personal strengths and big-picture pattern recognition. |
Statements from this episode (25)
Foody: AI evals are the product requirement documents for models
“If the model is the product, then the eval is the product requirement document.”
Foody: Success measurement bottlenecks economy-wide AI automation
“And so in many ways, the barrier to applying agents to the entire economy To automate every workflow is how do we measure success? How do we eval it and write the PRDs for everything that we want agents to do, which Mercore is obviously a huge part of doing.”
Foody: AI labs and apps will use evals as sales collateral
“I think labs will increasingly use labs as well as application layer companies will increasingly use evals to demonstrate the capabilities of their models and their products.”
Foody: Mercor bootstrapped to $1M run rate before founders dropped out
“Bootstrapped the company to a million dollar revenue run rate before we dropped out of college.”
Mercor scaled from $1M to $400M revenue run rate in 16 months
“We grew from one to four hundred million in revenue run rate in 16 months, and it's been an extraordinary journey and super exciting.”
Foody: Human AI data market is bound by human-model capability gap
“Effectively, the market is bound by the amount of things where humans can do something that models can't.”
Foody: AI evals and RL environments share the exact same data type
“There's not actually a nuance in the data type. It's more just a different semantic way of what describing what it's being used for. But ultimately it's just some stasis point for like, how do you measure what good looks like?”
Foody: AI labs are broadly shifting from RLHF to RLAIF
“What everyone is generally moving towards is reinforcement learning from AI feedback instead of human feedback, where you have instead the human defined some sort of success criteria, some way to measure that... And it's far more scalable and data efficient, a…”
Foody: Human AI eval work will last as long as human advantages
“And I think that that road to improving models will last for as long as there is anything in the economy that humans can do, which models can't, and be a huge portion of what the future of work looks like.”
Foody: The entire economy will likely become an RL environment machine
“It speaks to conversations I've had with a lot of researchers and executives at top labs, which is that it's highly likely that the entire economy will become an RL environment machine.”
Foody: Software development has virtually unlimited elastic demand
“Like in accounting, I think realistically we only need so much accounting in the world, right? Like maybe there's areas where we can do more and that'll be good. But it doesn't feel like the world needs a hundred times more accounting. On the other hand, in so…”
Foody: Some Fortune 500s fear evaluating AI automation in their businesses
“There are certain enterprises we talk to that are almost like fearful, not wanting to engage, not wanting to, you know, eval their businesses because that'll provide the evidence that their value chain is being automated. And there's others that, I mean, liter…”
Foody: Software automation will create a unified global labor market within 10 years
“When we're able to Automate that matching problem at the cost of software. It makes way for this global unified labor market that every candidate applies to and every company hires from facilitating a perfect flow of information in the economy. And I think tha…”
Foody: Tens of thousands work on AI post-training at any given time
“Tens of thousands at any given time, hundreds of thousands more generally. I mean, it's huge. And the most exciting thing is that it's growing really quickly.”
Foody: Traditional expert networks struggle to meet AI post-training demands
“One core difference is that alpha sites would generally be a one-off call versus a lot of our work is really hiring people for projects, right? Of how do they work on something for a longer period of time? And so that That's, I think, one of the reasons that s…”
Mercor hires Harvard Lampoon staff and Emmy winners to train AI
“Like we hired all the people from the Harvard Lampoon a couple of months ago, their comedy club to help with making models funnier. And so do all sorts of stuff like that, hiring Emmy award-winning screenwriters and everything across the board on creative capa…”
Foody: Top 10% of human evaluators drive majority of model improvement
“And there's also this really interesting dynamic where in a set of a hundred people that we hire, oftentimes the top 10% of people will drive majority of the model improvement.”
Mercor's median AI trainer pay is $95/hr, reaching up to $500/hr
“I mean, so our median pay rate in the marketplace is 95 dollars an hour, but it can flex up well up into like 500 dollars an hour based on the depth of someone's expertise.”
Foody: Traditional AI crowdsourcing platforms typically pay around $30 per hour
“If you look at the economics of the crowdsourcing companies, oftentimes they would pay like 30 dollars an hour to town as sort of the average.”
Mercor had no sales or marketing personnel for its first 18 months
“For the last year and a half of the business, we've had no one in sales and marketing.”
Foody: Mercor is lifetime profitable and has never burned net capital
“We bootstrapped the company to a million dollar revenue run rate and have always remained super capital efficient. Like we've never burned money. We were lifetime profitable.”
Mercor hit its $50M run rate forecast two weeks after pitching
“Where I remember when we were talking to Benchmark before they led our series A, we were at 1.5 million in run rate. And I said we'd be at fifty million in run rate by the end of the year. And they said we were absolutely insane, right? As anyone, anyone would…”
Foody: AI superintelligence is more than three years away
“Like, I don't think it's, I know there's been some executives at big labs that say we'll have super intelligence in three years, but I think the truth is that it's a longer road.”
Foody: AI will automate majority of knowledge work tasks in 10 years
“Like, I think we'll be able to automate a majority of knowledge work tasks in, in the next 10 years for sure.”
Foody: Efficient post-training datasets, not 10x pre-training, drive AI progress
“And it's not going to be, you know, 10 X more pre-training data that gets those capabilities. It's much more going to be all of the post-training data sets that are far more data efficient and thoughtful that help us get there.”