Q that you don't know that you don't know that is going to catch you off guard. Like some competitor is going to come in and eat your lunch or whatever, right? So to me, that is the, I think the debate of like why people are moving from offline to online or online to vibechecks is because it costs a lot of effort to do offline and it probably lasts
A Maybe six months if you're lucky. Well, I think, I think you're conflating two different things. One thing is sitting in a room and enumerating a set of scenarios or tests that you think represent a workload. And then the other thing is choosing to run those tests outside of production. And both of those things are activities that can happen offline. But I don't think anyone who's legit who's doing offline evals is doing the former anymore. Like, In fact, when we started BrainTrust and we talked to customers, we're, um, sort of very excited about BrainTrust helping them create golden data sets. And now I actually think that term is kind of a dirty term. People don't really want to create golden data sets. It's, I think it's, it's, it's often a wasted effort to, to the point that you're making. I think the best teams view offline evals as a mechanism of reconciling what they see in production with real users who are using the product with Tests and iteration that they can do offline. Like the best teams that we work with on a daily or even more frequently sometimes basis are discovering use cases from logs online and then pulling them into their environments and then playing with them. And I think the difference between doing offline evals and online evals or offline evals and just pure vibe checks is I think the pure vibe check or pure online version of this is you Observe some…
AI assessment note: “Well, I think, I think you're conflating two different things.”