AI Evals

“Really the hardest part of routing is building the evals and trying to determine in what places a set of intelligences should be used for a particular application.”

“Now, one thing you should know as a product manager is a lot of people go straight to this, like, agreement. They say, okay, my judge agrees with the human at some percentage of the time. Now that sounds appealing, but it's a very dangerous metric to use becau…”

“The top one is, hey, I can just buy a tool, plug it in, and it'll do the eval for you. Why do I have to worry about this? We live in the age of AI. Can't the AI just eval it? That's the most common misconception. And people want that so much that people do sel…”

“Usually I'll spend three to four days really working with whoever to do initial rounds of error analysis, like a lot of labeling, feel like we're in a good place to create the spreadsheet that Hamill had and everyone's kind of on board and convinced, and even …”

“As you're, you know, doing developing a model or you're developing research ideas, You have to have good evals, and usually the researchers are the ones building those, and I think that's kind of just the wrong model. For us, at least, designers, my point of v…”

“I think usually like small numbers of very high quality evals are the way to go and like we curate them manually and make sure they're really good.”

“All the things that go into making a good software engineer, that's gonna be really hard to do. Like, I think it's going to be a years long build out for even some of the verifiable domains. Cause there's so much that goes into a good software engineer of like…”

“The foundation of evals is error analysis. So like looking at your data and doing data analysis on your traces. So a lot of people, when we say data literacy, that can mean, that can sound scary, but it can come from a lot of different places. It can be, you c…”