Benchmark Contamination
topic on 1 show · 1 statements across 1 episodes
1 statements about Benchmark Contamination, every show
Stanford researchers detect benchmark contamination by testing evaluation question ordering
“There's a paper by Tatsu at a Stanford. Where they check if the order that the specific questions happen to be in matters. And if the answer is yes, then you probably trained on it because the order of the questions is arbitrary and shouldn't matter.”