AI agent evaluation

“And so I think like how quickly it's able to make progress is actually a pretty reason or a reasonable like eval if a slightly expensive one to calculate. It's an integration test, not a unit test.”

“When you run it for that long, you create so much data and to just say like, oh, the number is X. And then you throw away everything else. That's just very wasteful. There's so much insight from the things leading up to that number and reading the traces is li…”