LLM Evaluation
topic on 2 shows · 3 statements across 3 episodes
3 statements about LLM Evaluation, every show
Husain: Custom annotation web apps yield massive ROI for AI evals
“One counterintuitive thing that has an extreme value That people kind of discover maybe accidentally are, you know, if they're working with us, they discover very fast is Is this, there's a really, so you really want to look at your data a lot, and there's a r…”
Huang: Needle in a Haystack is a primitive LLM benchmark prerequisite
“I think needle in a haystack is definitely, like, the standard for presenting the work in a way that people can understand and also proving out. I would say, like, I view it as, like, a primitive. That you have to pass in order to give the model any shot of do…”
Alexandr Wang: Scale AI will launch recurring held-out LLM benchmark leaderboards.
“So one is that we're going to launch these private held out evaluations and have leaderboards associated with these evals for the leading LLMs in the ecosystem. And we're going to rerun this contest periodically. So every few months we're going to do a new set…”