LLM grader
2 statements across 1 episodes · 1 bullish · 1 bearish · 1 people on the record · first statement Jul 29, 2025 by Brendan Fortuna · across every show →
Everything said about LLM grader, oldest first
Jul 29, 2025 negative
Fortuna: LLM Graders for Prose Generation Are Highly Vulnerable to Reward Hacking
“And whenever using like an LLM grader, the task is like a little bit more pros or a little longer form generation. You could be very vulnerable to this. The models are super clever. They're incentivized to win, but they'll cheat and they'll do weird things.”
Jul 29, 2025 positive
Fortuna: Weighting Graders 75% Accuracy and 25% Style Mitigates Reward Hacking
“So what we did is in the grader, you know, in addition to just the content and like the semantic accuracy of what it's saying, we also started to add style. And we kind of weight them like 75, 25, and over time you can kind of harness and get the reward hackin…”