Offline Evals

“Offline evals, which I think is what a lot of the debate was about are both the most challenging to build feedback loop and also the most efficient once built. And then I think AB tests are a little bit less challenging to build and a little bit less efficient…”

“People don't really want to create golden data sets. It's, I think it's often a wasted effort to the point that you're making. I think the best teams view offline evals as a mechanism of reconciling what they see in production with real users who are using the…”

“Even if you get an offline 95, by the way, usually that means the product, when it gets to market, it's a really bad product, at least in AI world, in my opinion.”