AI benchmarks
2 statements across 2 episodes · 0 bullish · 1 bearish · 2 people on the record · first statement Dec 11, 2024 by Alexandr Wang · across every show →
Everything said about AI benchmarks, oldest first
Dec 11, 2024 neutral
Wang: AI benchmarks are saturated, making model leaders hard to distinguish
“One thing that we see today with the models is that because all the benchmarks that were used today are what's called saturated, i.e., you know, in other words, like all the models do really well at the benchmarks, it's really hard to discern actually which on…”
May 14, 2025 negative
Bender says most AI benchmarks fail to measure actual capabilities
“Most of the benchmarks that are out there are not reasonable. They lack what's called construct validity, and construct validity is this two-part test of the thing that we are trying to measure is a real thing, and this measurement correlates with it interesti…”