Dec 7, 2025 · 34m · latent-space
The Great Evals Debate — Ankur Goyal & Malte Ubl
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space Lightning Pod, Swyx hosts Braintrust founder Ankur Goyal and Vercel CTO Malte Ubl to debate the transition from 'vibe coding' to robust evaluation methodologies for AI coding agents. They delve into production-driven feedback loops, reinforcement learning pipelines, and the strategic role of open-source framework benchmarks.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 23.2% of the talking time here. How this is scored →
The hosts as informed peer 4.6 Guest teaching 4.4 Guest disagreement 3.3 The hosts pushing back 3.7
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 9:40 Ankur rejects Swyx's premise on offline evals
Ankur firmly tells Swyx he is conflating offline evals with creating static golden datasets in a room, rejecting Swyx's argument that open-endedness makes offline testing obsolete.
Hardest push from the hosts ▶ 22:11 Swyx rejects broadening 'evals' to include vibesSwyx directly challenges Ankur's attempt to classify vibes under evals, arguing that if vibes count, the definition loses meaning because vendor bias makes everything look like an eval.
Biggest teaching moment ▶ 9:55 Ankur explains modern production log replay over golden datasetsAnkur educates Swyx on how top engineering teams actually run offline evals by pulling live failures from prod logs with one click rather than authoring brittle golden test suites.
The host holds their own ▶ 25:15 Swyx formulates the scaling thesis for synthetic RL environmentsSwyx demonstrates domain expertise by framing RL environments in computer use as encoded human intuition designed to break past human throughput bottlenecks.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Defining Evals, Vibes, and Feedback Loops in AI | 2 | 3 | 1 | 1 | Swyx sets the stage by bringing up Boris Cherny's vibe coding comments, after which Malte and Ankur give structured opening philosophies on feedback loops and why vibe checks scale poorly compared to automated evals. The dynamic is collaborative with the host mostly listening. | |
| RL Loops, AI Lab Privilege, and Public Benchmarks | 5 | 4 | 3 | 3 | Malte notes that frontier agent teams benefit from unfair lab proximity. Swyx counters with real-world agent bench burnout at Cognition, prompting Ankur to clearly delineate public marketing benchmarks from internal iteration feedback loops. | |
| Offline Evals as Production Replay vs Brittle Unit Tests | 4 | 7 | 5 | 4 | Swyx challenges offline evals by claiming open-ended coding outgrows static test suites quickly. Ankur directly pushes back, telling Swyx he is conflating offline running with static golden datasets and explaining modern production replay workflows. | |
| Vercel's Practical Eval Workflow and Composite Models | 5 | 5 | 4 | 4 | Malte outlines Vercel's composite model approach to fix trivial syntax errors fast. Swyx tries to summarize it as avoiding agentic loops entirely, which Malte immediately corrects before Swyx cites RL turn optimization from Cognition's SWEGREP work. | |
| Evals as Product Specs and Domain Knowledge Transfer | 5 | 4 | 4 | 6 | Ankur describes how evals replace 50-page PRDs for product managers. When Ankur suggests vibes are just another form of eval, Swyx pushes back against diluting the term, arguing that calling everything an eval makes the definition meaningless. | |
| Reinforcement Learning Environments and Synthetic Feedback | 6 | 3 | 3 | 5 | Ankur expresses skepticism that average companies can build RL environments without reward hacking. Swyx jumps in with a strong conceptual counterpoint, arguing RL environments scale because they encode domain rules to decouple learning from slow human feedback. | |
| Inversion of Control: Publishing Evals for Model Labs | 5 | 5 | 3 | 3 | Malte proposes an inversion of control where framework authors publish evals to force model labs to optimize for them. When Swyx compares this to independent rating agencies like LMSYS, Malte clarifies the self-interested incentive model behind Vercel's strategy. |
Statements from this episode (15)
Ubl: Evals Function to Tell Developers Overnight Whether a Change Is Good
“The way I think about evals is essentially like, it's the thing that, that can tell me tomorrow whether my change is good. And I can operate without that knowledge, but it's super, super helpful.”
Goyal: Offline Evals Are Hardest to Build but Most Efficient Feedback Loop
“Offline evals, which I think is what a lot of the debate was about are both the most challenging to build feedback loop and also the most efficient once built. And then I think AB tests are a little bit less challenging to build and a little bit less efficient…”
Goyal: AI Coding Has the Most Product Market Fit After ChatGPT
“The first is it is the use case that has the most product market fit in AI, I think, other than ChatGPT.”
Ubl: Cognition and Cursor shipped RL fine-tunes of open-source models
“Just yesterday, I think we saw both Cognition, congrats, SWIX, and Cursor to ship RL fine tunes of unnamed open source models.”
Ubl: Anthropic's Boris Power Vibe-Codes From a Position of Privilege
“I think he comes from a particular position of extreme unusual privilege, which is that he works at an AI lab where like people in the office next door are like writing the evals and are like training the model like every day in exactly that way.”
Goyal: Publishing Public Benchmarks Is Marketing, Not Product Improvement
“It's just that the value proposition of publishing an eval is completely orthogonal to the value proposition of building evals in service of building a good product. I think the purpose of publishing benchmarks is marketing, and it's good marketing.”
Goyal: Creating Golden Datasets for AI Evals Is Wasted Effort
“People don't really want to create golden data sets. It's, I think it's often a wasted effort to the point that you're making. I think the best teams view offline evals as a mechanism of reconciling what they see in production with real users who are using the…”
Malte Ubl: When Vibes and Eval Data Disagree, Vibes Are Right
“I think that the common quip that if the vibes and the data disagree, the vibes are probably right. It's true, right? So you have to like, be honest with yourself, like, do they agree and kind of iterate On them over time.”
Goyal: North Star AI Evals Prevent Test Brittleness
“Like if you construct evals in a way that represent the true north star of the problem that you're solving, then they tend not to break as you change the underlying system. Whereas if you hard code them to a narrow subset of like an implementation detail of yo…”
Ubl: Vercel's Composite Models Are Faster Than Agentic Loops
“Basically what we do is we have this like composite model architecture. We run the frontier model and then we run the fine tune model after to fix its errors. That doesn't perform better than an agentic loop, but it's orders of magnitude faster, right?”
Goyal: Braintrust sees surge in PMs and designers joining eval process
“We've seen like a massive surge of product manager, product managers and designers getting interested in participating in the eval process among our customers.”
Goyal: Providing eval criteria and examples is more effective than writing specs
“In many ways coming to the table of product building with representative examples and criteria that articulate what good versus bad is for a use case is just a more precise and usable form of product management than writing a spec.”
Goyal: Average AI companies cannot hire expertise to prevent reward hacking
“You need to have like a pretty specific expertise to design the RL environment in a way that's not vulnerable to reward hacking. And I think that either you'll end up with some fixed number of very well engineered RL environments, or you need to somehow employ…”