Real World Evaluation
topic on 1 show · 1 statements across 1 episodes
1 statements about Real World Evaluation, every show
Narayanan: AI developers over-optimize models for benchmarks over real-world performance
“When there is so much pressure to do well on these benchmarks, developers are intentionally or unintentionally optimizing these models In ways that look good on the benchmarks, but don't look good in real world evaluation.”