LLM as a judge

“I've inspected the outputs of these and like, they're almost always correct. If you sort of, if you ask the model to judge these things in the right way, they're very good at being able to tell this.”

“I think when we simplify it to binary classification metrics, I think it can work. And I think a lot of things can be simplified, like Shreya mentioned, I think a lot of things can be simplified to binary classification metrics. And I've seen evidence of it wo…”

“We already have a bunch of open models that are doing like judge of models and Prometheus and other things that are designed specifically for LM as a judge. And I see that continuing to just become part of this kind of open RL infrastructure.”

“I feel like a 75% LMS judge because it's low effort is kind of easy, but I would say out of the 75%, 80% is not helpful.”

“I put very little value in benchmarks, like general benchmarks. It's has some value, but you know, what you really need to do is like measure it in your domain and see if that alum as a judge is more aligned than like an off the shelf LLM. And what I've found …”

“One really huge thing about LLM as a judge, people love LLM as a judge, but you really have to make sure that you can trust the LLM as a judge. And so how do you trust and how do you trust anything is that you have to check it and you have to measure how good …”

“They initially started with five LLM as judges. So each one of these points had their own LLM as a judge. They tested the ability and accuracy of that LLM as judge collective to judge, and it actually didn't perform as well as one. So they replaced all of thos…”

“The leaderboard, if you are using different judges with different models, it can, there can be heavy shakeup of the leaderboard also.”

“If you ask that model, try to like use it as an LLM as a judge, it's not very good. But if you probe its mind and you sort of detect when the features related to personally identifiable information are firing, that gets you the highest recall of anything. It's…”