Soldani: Ai2 Trains OLMo Using Community Datasets and Other Models' Outputs
Luca Soldani · Best of 2024: Open Models [LS LIVE! at NeurIPS 2024] · Dec 23, 2024 · at 4:55
Luca Soldani of the Allen Institute for AI (Ai2) explains how the open-source AI community accelerates research by sharing datasets and model outputs during training.
0:00 / 0:37exact quote · 37.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We see a lot of these even in our own work of like, you know, as we iterate in the various version of Olmo it's not just like every time we collect from scratch all the data. No, the first step is like, okay, what are the cool data sources and datasets people have put together for language model for training? Or when it comes to like our post training pipeline we one of the steps is You want to do some DPO, and you use a lot of outputs of other models to improve your preference model.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Luca Soldani
Soldani: Open Model Bio-Risk Warnings Were a Lobbying Ploy
“You know, if you remember the beginning of this year, it was all about bio-risk of these open models. The whole thing fizzled out because there's been, finally there's been, like, rigorous research, not just this paper from coherent folks, but there's been rig…”
Soldani: Frontier LLM pre-training requires at least 50,000 GPUs
“To give you a sense of, like, how I personally think about research budget for each part of the language model pipeline is, like, on the pre-training side, you can maybe do something with a thousand GPUs. Really, you want 10,000. And, like, if you want real es…”
Soldani: Replicating OpenAI's o1 requires roughly 10,000 GPUs
“If you're interested in you know, your, Open replication of what OpenAI's O-one is you're gonna be on the 10 K spectrum of our GPUs.”
Soldani: Web blocking disproportionately benefits incumbent closed AI labs
“And I think the problem is this blocking or ideas really, it impacts people in different ways. It disproportionately helps companies that have a head start, which are usually the closed labs, and it hurts incoming newcomer players where you either have now to …”
Soldani: Local models beat closed models in retrieval applications
“There are some applications where local models just blow closed models out of the water. So, like, retrieval is a very clear example.”
Soldani: Llama and Qwen models fail OSI open source AI definition
“Under this definition, for example, Lama or some of the Quen models are not open source because the license says you can, you can't use this model for this, or it says if you use this model, you have to name the output this way or derivative needs to be named …”