LLM evaluation
2 statements across 2 episodes · 1 bullish · 0 bearish · 2 people on the record · first statement May 31, 2024 by Mark Huang · across every show →
Everything said about LLM evaluation, oldest first
May 31, 2024 neutral
Huang: Needle in a Haystack is a primitive LLM benchmark prerequisite
“I think needle in a haystack is definitely, like, the standard for presenting the work in a way that people can understand and also proving out. I would say, like, I view it as, like, a primitive. That you have to pass in order to give the model any shot of do…”
Mar 13, 2025 positive
Husain: Custom annotation web apps yield massive ROI for AI evals
“One counterintuitive thing that has an extreme value That people kind of discover maybe accidentally are, you know, if they're working with us, they discover very fast is Is this, there's a really, so you really want to look at your data a lot, and there's a r…”