MLE Bench
product on 2 shows · 5 statements across 1 episodes · said 4 times in 3 episodes since 2024
Latent Space 3 
the Neon Show 1
Mentions by year, every show
tap a year for its mentions
Latent Space 3the Neon Show 1
2026 1 mention in 1 episode
2025 1 mention in 1 episode
2024 2 mentions in 1 episode
every mention on every show, scene by scene, with the transcript →
5 statements about MLE Bench, every show
Hu: OpenAI o1-preview achieves bronze medals in 17% of MLE-bench competitions
“Their final results with a one preview and this a scaffolding from a different company was that they got a bronze medal. I don't think I've ever achieved once but I haven't competed that much in. 17% of competitions.”
Hu: OpenAI o1-preview surpasses human Kaggle Grandmasters with seven gold medals
“Since a grandmaster requires five gold medals and oh, and preview gets an average of eight or sorry, seven gold medals. They're out competing even capital grandmasters.”
Hu: GPU setups showed virtually no agent performance gain over CPU-only
“They compared a CPU only setup to a GPU setup to a multi GPU setup, and it kind of made no difference really.”
Hu: Single MLE-bench evaluation run with OpenAI o1-preview costs $4,000
“Just for one seed, For one run of these things cost 4000 dollars all in with the GPU plus the tokens. And a bulk of the cost was actually the token, so even if you cut the GPU out, it'll still cost you three grand to run on one preview.”
Hu: MLE-bench authors found obfuscating competition details did not show overfitting
“They do a lot of checks against overfitting on the Kaggle tasks themselves, and so they do something where they obfuscate some of the details of the Of the competitions, and then they rerun it. And I guess if they were overfitting on the competitions themselve…”