Eval Datasets
topic on 1 show · 2 statements across 1 episodes
2 statements about Eval Datasets, every show
Bhatawdekar: Production eval datasets must be continuously updated from live logs
“These eval datasets, they are not static. As the teams look at their logs, at how their systems are working in the real world, in the production use cases, they're able to leverage those insights to continually augment their eval datasets. And so these eval da…”
Bhatawdekar: Notion, Stripe, and Zapier run automated evals on every system modification
“All of these companies are building agents, intelligent systems that are performing specialized tasks for whatever use cases they have, and so all of them build evals that reflect how their systems are expected to behave in production. They're all following si…”