Private Evals
topic on 2 shows · 6 statements across 2 episodes
6 statements about Private Evals, every show
Nadella: Public AI benchmarks are gamed; companies need private evaluations
“Most importantly, you'll have private evals because we know all the evals out there are good, interesting, But they're not really that critical at this point because they're all can be maxed. And so the point is each company will have its own private eval.”
Nadella: Private AI evals may be an enterprise's biggest IP
“Every company having private evals may be the biggest IP, right? I think about it, like what's that private eval that you can then use even a frontier model to hill climb on and not leak the traces. Maybe one of the biggest drivers of IP.”
Nadella: Model swappability via private evals tests true AI control
“Another acid test is you have an eval that's private. You're using a model A. Can you switch it to model B and, you know, climb up? If you can, then you're in control. If you can't, you're not in control.”
Nadella: Public AI evals are maxed out; every company will use private evals
“Most importantly, you'll have private evals, because we know all the evals out there are good, interesting, but they're not really that critical. At this point, because they're all can be maxed. And so the point is each company will have its own private eval.”
Nadella says private evals and execution traces are a company's biggest IP
“So that's why I would say every company having private evals may be the biggest IP, right? I think about it like what's that private eval that you can then use even a frontier model to hill climb on and not leak the traces. Maybe one of the biggest drivers of …”
Nadella: Enterprise AI control requires private evals to switch underlying models safely
“Like, so in other words, another asset test is, you have an eval that's private. You're using a model A. Can you switch it to model B and, you know, climb up? If you can, then you're in control. If you can't, you're not in control.”