academic benchmarks
2 statements across 2 episodes · 0 bullish · 2 bearish · 1 people on the record · first statement May 22, 2024 by Alexandr Wang · across every show →
Everything said about academic benchmarks, oldest first
May 22, 2024 negative
Alexandr Wang: Academic AI benchmarks are contaminated and models are overfit.
“The academic benchmarks that are what the industry used to measure the performance of these algorithms are fraught with issues. Many of the models are overfit on these benchmarks. They're sort of in the training data sets of these models.”
Jul 11, 2024 negative
Wang: Standard academic benchmarks are contaminated by training data overfitting
“Most of the benchmarks that we as a community look at... Academic benchmarks that are what the industry used to measure the performance of these algorithms are fraught with issues. Many of the models are overfit on these benchmarks. They're sort of in the trai…”