evals
4 statements across 3 episodes · 3 bullish · 0 bearish · 4 people on the record · first statement May 30, 2025 by Garry Tan · across every show →
Everything said about evals, oldest first
May 30, 2025 positive
Garry Tan: Vertical AI moats require sitting with domain workers to build evals
“You can't get the evals unless you are sitting literally side by side with people who are doing X, Y, or Z knowledge work. You know, you need to sit next to the tractor sales regional manager and understand, well, you know, this person cares about, you know, t…”
May 30, 2025 positive
Jared Friedman: Evals, not prompts, are the core data asset for AI startups
“Even though we've been saying this for a year or more now, Gary, I think it's still the case that like evals are the true crown jewel Like data asset for all of these companies. Like one reason that power help was willing to open source the prompt is they told…”
Feb 6, 2026 positive
Jul 27, 2026 neutral
Cherny: AI evals saturate and must be discarded every few generations
“I think evals, they outlive the harness a little bit, but not quite that much. Like, an eval might live for maybe one, two, three model generations, but nowadays the, you know, we're on the exponential. The model is improving so quickly, very often we just sat…”