benchmark contamination
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Aug 28, 2024 by Nicholas Carlini · across every show →
Everything said about benchmark contamination, oldest first
Aug 28, 2024 neutral
Stanford researchers detect benchmark contamination by testing evaluation question ordering
“There's a paper by Tatsu at a Stanford. Where they check if the order that the specific questions happen to be in matters. And if the answer is yes, then you probably trained on it because the order of the questions is arbitrary and shouldn't matter.”