General Benchmarks
topic on 2 shows · 2 statements across 2 episodes
2 statements about General Benchmarks, every show
Husain: General LLM benchmarks do not correlate with product-specific evals
“Up until now, a lot of the big labs understandably focused on general benchmarks, like MMLU score, human eval, things like that, which are very important for foundation models. And, you know, those not very related to product specific evals, like the ones we t…”
Husain: General benchmarks for LLM judges provide very little value
“I put very little value in benchmarks, like general benchmarks. It's has some value, but you know, what you really need to do is like measure it in your domain and see if that alum as a judge is more aligned than like an off the shelf LLM. And what I've found …”