AI Agent Evaluation

“Like I see companies like measuring number of calls or minutes during the call or some like superficial KPIs that give you some information, but that doesn't really work. Like the important thing is Did this customer convert? Is it bringing value to the custom…”

“When you run it for that long, you create so much data and to just say like, oh, the number is X. And then you throw away everything else. That's just very wasteful. There's so much insight from the things leading up to that number and reading the traces is li…”

“And so I think like how quickly it's able to make progress is actually a pretty reason or a reasonable like eval if a slightly expensive one to calculate. It's an integration test, not a unit test.”