Academic Benchmarks
topic on 2 shows · 3 statements across 3 episodes
3 statements about Academic Benchmarks, every show
Pineau: Selling enterprise AI offers better feedback than academic benchmarks
“When you need to sell AI to a business, you get a real signal of what works, what doesn't work. And you know, we've been using these academic benchmarks for many years. You get some signal, but it's not the same as getting this to do productive work.”
Wang: Standard academic benchmarks are contaminated by training data overfitting
“Most of the benchmarks that we as a community look at... Academic benchmarks that are what the industry used to measure the performance of these algorithms are fraught with issues. Many of the models are overfit on these benchmarks. They're sort of in the trai…”
Alexandr Wang: Academic AI benchmarks are contaminated and models are overfit.
“The academic benchmarks that are what the industry used to measure the performance of these algorithms are fraught with issues. Many of the models are overfit on these benchmarks. They're sort of in the training data sets of these models.”