Dec 7, 2025 · 1h 10m · lennys-podcast
The $1B Al company training ChatGPT, Claude & Gemini on the path to responsible AGI | Edwin Chen
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Surge AI founder and CEO Edwin Chen discusses how his company bootstrapped to over one billion dollars in revenue by rethinking AI data quality, rejecting venture capital conventions, and prioritizing human taste over engagement-driven metrics.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 30.9% of the talking time here. How this is scored →
Lenny as informed peer 3.5 Guest teaching 5.0 Guest disagreement 3.1 Lenny pushing back 0.8
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 23:30 Denouncing LM Arena as tabloid optimization
Edwin forcefully attacks standard industry leaderboards like LM Arena, calling them superficial dopamine-chasers equivalent to grocery store tabloids.
Hardest push from Lenny ▶ 27:37 Lenny steelmanning entertainment models like SoraLenny counters Edwin's skepticism of frivolous models by steelmanning OpenAI Sora as enjoyable, revenue-generating, and valuable for video training.
Biggest teaching moment ▶ 9:59 Explaining the difference between metric-checking and real qualityEdwin completely reframes data annotation from counting lines in a poem to evaluating Nobel-prize-caliber artistic depth and emotion.
Lenny holds their own ▶ 30:52 Lenny citing his research on early employees at generational startupsLenny demonstrates his domain expertise by validating Edwin's anti-pivot philosophy using qualitative data from his study on early employees at OpenAI and Stripe.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| Bootstrapping Surge AI to a Billion-Dollar Run Rate | 3 | 5 | 4 | 1 | Lenny opens by admiring Surge AI's unprecedented revenue-to-employee ratio and asks how Edwin leveraged AI. Edwin explains his contrarian philosophy of firing 90% of big tech workers and avoiding the Silicon Valley PR hamster wheel. | |
| Defining Data Quality and Annotation Methodologies | 4 | 6 | 3 | 1 | Lenny asks how data quality is defined and measured. Edwin dismantles simplistic checkbox approaches, using poetry and Google search ranking analogies to explain how Surge captures nuanced qualitative signals. | |
| The Art of Post-Training and Model Taste | 4 | 5 | 2 | 0 | Lenny inquires why Claude was superior at writing and coding for so long. Edwin explains that post-training is an art involving human taste, aesthetic tradeoffs, and prioritizing real tasks over marketing benchmarks. | |
| The Pitfalls of Benchmarks and Accurate Evals | 3 | 7 | 5 | 1 | Lenny asks if academic benchmarks correlate with real-world AI capabilities. Edwin strongly dismisses standard benchmarks as flawed and easily gamed, noting models can win IMO math medals but struggle to parse basic PDFs. | |
| The Risks of AI Slop and Engagement Optimization | 4 | 6 | 6 | 3 | Edwin launches a fierce critique of LMSYS Chatbot Arena leaderboards and social-media-style engagement optimization, calling it tabloid baiting. Lenny steelmans commercial video models like Sora, prompting Edwin to reiterate the dangers of deceptive shortcuts. | |
| Contrarian Startup Strategy and Resisting the Pivot | 5 | 4 | 5 | 1 | Edwin passionately denounces the Silicon Valley playbook of rapid pivoting, blitzscaling, and prestige hiring. Lenny validates Edwin's perspective by sharing findings from his own research with VC Terence Rohan on generational companies. | |
| Sponsor Break: Coda | 4 | 6 | 2 | 1 | Following the Coda sponsor segment, Lenny brings up Richard Sutton's Bitter Lesson and RL post-training environments. Edwin explains how reinforcement learning environments simulate real multi-step tasks rather than isolated academic tests. | |
| Surge AI's Research Lab Culture and Hiring Standards | 3 | 5 | 2 | 0 | Lenny asks why Surge invests in internal research teams unlike most data vendors. Edwin explains his desire to advance the scientific frontier rather than focus solely on valuation, noting he would rather be mathematician Terence Tao than Warren Buffett. | |
| Model Differentiation, Mini-Apps, and Vibe Coding | 4 | 4 | 3 | 1 | Edwin shares how corporate values will create differentiated model personalities and highlights interactive chatbot mini-apps as underhyped while calling vibe coding overhyped. Lenny connects this to previous discussions with product leaders. | |
| Origins of Surge AI and Edwin Chen's Career | 3 | 4 | 1 | 0 | Lenny compares Edwin's interdisciplinary background to Coinbase founder Brian Armstrong. Edwin recounts his journey through MIT linguistics, big tech research frustration with simple image labeling, and launching Surge AI right after GPT-3. | |
| The Philosophy of Training AI Like Raising a Child | 2 | 5 | 3 | 0 | Edwin reflects philosophically on AI training, comparing it to raising a child and teaching values rather than optimizing for simplistic proxy metrics. Lenny expresses appreciation for Edwin's deep philosophical framing. | |
| Lightning Round: Recommendations, Sci-Fi, and Soda vs. Pop | 3 | 3 | 1 | 0 | In the lightning round, Edwin shares favorite sci-fi books about linguistics, praises Waymo, and jokes about his viral Twitter soda versus pop dialect map. |
Statements from this episode (15)
Surge AI Surpassed $1B in Revenue With Under 100 Employees
“Yeah, so we hit over a billion of revenue last year with under a hundred people.”
Chen: AI Efficiency Will Enable $100 Billion Revenue-Per-Employee Ratios
“And I think we're going to see companies with even crazier ratios, like a hundred billion per employee in the next few years. AI is just going to get better and better and make things more efficient. So that ratio just becomes inevitable.”
Chen: Big Tech Could Fire 90% of Staff and Move Faster
“Like I used to work at a bunch of the big tech companies, and I always felt that we could fire 90% of people and we would move faster because the best people wouldn't have all these distractions.”
Chen: Throwing bodies at data labeling fails to create quality AI data
“I think most people don't understand what quality even means in this space. They think you can just throw bodies at a problem and get good data, and that's completely wrong.”
Surge AI tracks keystrokes and model gains to evaluate data annotators
“The way it works is we essentially gather thousands of signals about everything that you're doing when you're working on a platform. So we are looking at your keyboard strokes. We are looking how fast you answer things. We are using reviews. We are using code …”
Chen: AI post-training is an art driven by taste, not pure science
“One of the things I often think about is that there's a, it's almost like there's an art to post training. It's not purely a science. Like when you were deciding what kind of model you're trying to create and what it's good at. There's this notion of taste and…”
Chen: Public AI Benchmarks Are Unreliable and Often Contain Wrong Answers
“I don't trust the benchmarks at all. And I think that's for two reasons. So one is, I think a lot of people don't realize, even researchers within the community, they don't realize that the benchmarks themselves are often honestly just wrong. Like they have wr…”
Chen: Frontier AI Labs Game Benchmarks via Prompt Tweaking and Test Leaks
“Sometimes, yeah, these benchmarks, they accidentally leak in certain ways, or the frontier labs will tweak the way they evaluate their models on these benchmarks. Like they'll tweak their system prompt. Or they'll tweak the number of times they run their model…”
Chen: AI Will Automate 80% of L6 Engineer Tasks Within 2 Years
“In my head, I probably bet that within the next one or two years, yeah, the models are going to automate 80% of, you know, the average L six software engineer's job. But it's going to take another few years, do you move to 90%, and another few years to 99%, an…”
Chen: LMSYS Chatbot Arena rewards bolding, emojis, and length over accuracy
“The easiest way to climb Alamarina, it's adding crazy boating. It's doubling the number of emojis. It's tripling the length of your model responses. Even if your model starts hallucinating and getting the answer completely wrong.”
Chen: Anthropic stands out among frontier labs for principled model development
“I would say I've always been very, very impressed by Anthropic. Like, I think Anthropic takes a very principled view about what they do and don't care about. And how they want their models to behave in a way that feels a lot more principle to me.”
Chen: Silicon Valley is just as money-obsessed as Wall Street
“Silicon Valley loves to score on Wall Street for focusing on money. But honestly, most of the Silicon Valley is chasing the same thing.”
Chen: Achieving AGI will require breakthroughs beyond standard LLMs
“I'm in a camp where I do believe that something new will be needed.”
Chen predicts AI models will become increasingly differentiated across creator labs
“I think one of the things that's going to happen in the next few years is that the models are actually going to become increasingly differentiated because of the personalities and behaviors
That the different labs have and the kind of objective functions that …”