AI benchmarks
2 statements across 2 episodes · 1 bullish · 1 bearish · 2 people on the record · first statement May 16, 2024 by Mikey Shulman · across every show →
Everything said about AI benchmarks, oldest first
May 16, 2024 negative
AI evaluation benchmarks fail for audio, necessitating human aesthetic judgment
“I think in, in all branches of AI, we become slaves to our metrics, and you say, I did this accuracy on this benchmark, and this accuracy on this benchmark, and in the real world, sometimes it doesn't necessarily matter, and these benchmarks are extra terrible…”
Jun 26, 2026 positive
Brown: AI benchmarks must control for test-time compute
“And so I think the proper way to, and so my claim is the proper way to evaluate the models now is you either have some kind of budget for the benchmark, whether it's tokens or cost or time or whatever, or you plot the performance as a function of the amount of…”