The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 5 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Hu: OpenAI o1-preview surpasses human Kaggle Grandmasters with seven gold medals
“Since a grandmaster requires five gold medals and oh, and preview gets an average of eight or sorry, seven gold medals. They're out competing even capital grandmasters.”
Hu: Single MLE-bench evaluation run with OpenAI o1-preview costs $4,000
“Just for one seed, For one run of these things cost 4000 dollars all in with the GPU plus the tokens. And a bulk of the cost was actually the token, so even if you cut the GPU out, it'll still cost you three grand to run on one preview.”
Hu: MLE-bench authors found obfuscating competition details did not show overfitting
“They do a lot of checks against overfitting on the Kaggle tasks themselves, and so they do something where they obfuscate some of the details of the Of the competitions, and then they rerun it. And I guess if they were overfitting on the competitions themselve…”