real world evaluation
1 statements across 1 episodes · 0 bullish · 1 bearish · 1 people on the record · first statement Aug 28, 2024 by Arvind Narayanan · across every show →
Everything said about real world evaluation, oldest first
Aug 28, 2024 negative
Narayanan: AI developers over-optimize models for benchmarks over real-world performance
“When there is so much pressure to do well on these benchmarks, developers are intentionally or unintentionally optimizing these models In ways that look good on the benchmarks, but don't look good in real world evaluation.”