evals

“In AI, the whole point of observability is to collect data into data sets that you can use to do evals, and then again, eventually fine tune models or, you know, more advanced things.”

“Reinforcement learning is becoming so effective that once you create evals, the models can learn them and how to you know, improve capabilities. And so for everything that we want alums to be good at, we need evals for those things.”

“It Would not surprise me if that becomes the most common knowledge work job in the world.”

“It does seem structurally more efficient for work to trend away from the variable cost of like doing it repeatedly towards this fixed cost of how do we build out the evals and the processes for models to do this themselves.”

“You don't even know that you have super intelligence without having evals for everything. Cause it's like, you sort of need to understand what is the human baseline and like, what is good? It's like grounded in this like understanding of human behavior.”

“When you look at the model card for, let's say, the O-one paper that came out, I think, last year. If you look at the distribution of what most people worked on, on that paper, it was evals.”

“The single biggest differentiator that I see in the market is, does the team know how to drive a systematic error analysis process with evals? So you're building the agents by analyzing at any moment in time, what's working, what's not working, what do you imp…”

“Most importantly, you'll have private evals because we know all the evals out there are good, interesting, But they're not really that critical at this point because they're all can be maxed. And so the point is each company will have its own private eval.”

“I think it's a nice eval because there is very little open source code for making poker bots. And there's a lot of published essays, there's a lot of published papers on it, but you really have to reason through everything.”