Dec 23, 2024 · 37m · latent-space
Best of 2024: Open Models [LS LIVE! at NeurIPS 2024]
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Presented at NeurIPS 2024, this session reviews the major milestones, technical definitions, and systemic challenges facing open language models in 2024. The presentation features an in-depth analysis of fully open research pipelines by Ai2's Luca followed by an overview and live demonstration of Mistral AI's model portfolio and Le Chat platform.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 0.0 Guest teaching 0.0 Guest disagreement 0.6 The hosts pushing back 0.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 22:15 Calling out safety lobbying ploys
Luca forcefully dismisses alarmist bio-risk rhetoric as an ingenuous lobbying tactic designed to make AI sound terrifying and protect incumbent labs.
Hardest push from the hosts ▶ 24:50 Audience member challenges economic alignmentAn audience member presses on the core friction of open models by demanding how to sustainably align financial incentives rather than relying on goodwill.
Biggest teaching moment ▶ 7:20 Dissecting the flaws of the OSI AI definitionLuca breaks down the legal and technical ambiguity in OSI's open data requirement, educating listeners on why 'sufficiently detailed' data recipes fall short of genuine open source.
The host holds their own ▶ 25:04 Luca explains local vs global economic maximaLuca demonstrates deep domain insight by explaining why commercial actors only release open weights when it optimizes their local position on the market rather than the global ecosystem.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| The Rapid Evolution of Open Models in 2024 | 0 | 0 | 0 | 0 | Luca Soldani presents a monologue reviewing the state and progress of open models in 2024 compared to 2023. The host is not actively participating on mic. | |
| Collaborative Ecosystem and the Open Source Flywheel | 0 | 0 | 1 | 0 | Luca details the collaborative open source ecosystem and critiques the OSI open source AI definition for its weak data transparency requirements. Host does not intervene. | |
| The Rising Compute Divide and Training Recipes | 0 | 0 | 0 | 0 | Luca outlines the rising compute divide between frontier labs and researchers, breaking down GPU requirements across pre-training, post-training, and evaluation in a monologue format. | |
| The Principles and Power of Fully Open AI Models | 0 | 0 | 0 | 0 | Luca defines fully open models as releasing data, code, logs, and checkpoints rather than just final weights, giving examples of community replication. | |
| Ai2's 2024 Fully Open Model Releases and Recipes | 0 | 0 | 1 | 0 | Luca highlights Ai2 releases (OLMoE, MolMO, Tulu 3) and warns about diminishing open web training data caused by blanket crawling blocks and CDNs. Host does not participate. | |
| AI Safety Narratives, Policy, and Legislative Threats | 0 | 0 | 3 | 0 | Luca takes a critical stance against alarmist AI safety lobbying, debunking exaggerated bio-risk narratives and reflecting on California's SB 1047. | |
| Q&A: Structuring Incentives for Sustainable Open Models | 0 | 0 | 0 | 0 | An audience member inquires about structuring community incentives for sustainable open source development, and Luca provides a measured response on commercial vs. research motives. | |
| Mistral AI Overview: Model Portfolio and Roadmap | 0 | 0 | 0 | 0 | Mistral AI presenter delivers a monologue presentation cataloging Mistral's model family history and licensing options. No host interaction occurs. | |
| Live Demo of Le Chat: OCR, Canvas, and Multimodal Features | 0 | 0 | 0 | 0 | Mistral AI presenter walks through a live demo of Le Chat features including receipt OCR, Python browser execution in Canvas, and Tetris game generation. |
Statements from this episode (16)
Soldani: 2024 open models rival frontier performance of closed models
“You have models that are, you know, reveling frontier level performance of what you can get from closed models, from like Quen, from Deep Seek. We got Lama III.”
Soldani: Mechanistic interpretability research is impossible without open models
“There is a large swath of research on modeling, on how these models behave, on evaluation, on inference, on mechanistic interpretability that could not happen at all. If you didn't have open models.”
Soldani: Local models beat closed models in retrieval applications
“There are some applications where local models just blow closed models out of the water. So, like, retrieval is a very clear example.”
Soldani: Ai2 Trains OLMo Using Community Datasets and Other Models' Outputs
“We see a lot of these even in our own work of like, you know, as we iterate in the various version of Olmo it's not just like every time we collect from scratch all the data. No, the first step is like, okay, what are the cool data sources and datasets people …”
Soldani: Llama and Qwen models fail OSI open source AI definition
“Under this definition, for example, Lama or some of the Quen models are not open source because the license says you can, you can't use this model for this, or it says if you use this model, you have to name the output this way or derivative needs to be named …”
Soldani: Frontier LLM pre-training requires at least 50,000 GPUs
“To give you a sense of, like, how I personally think about research budget for each part of the language model pipeline is, like, on the pre-training side, you can maybe do something with a thousand GPUs. Really, you want 10,000. And, like, if you want real es…”
Soldani: Replicating OpenAI's o1 requires roughly 10,000 GPUs
“If you're interested in you know, your, Open replication of what OpenAI's O-one is you're gonna be on the 10 K spectrum of our GPUs.”
Soldani: Ai2 Released Data, Code, Logs, and All Checkpoints for MoE Model
“So I pull up the screenshot from our recent MOE model and, like, for this model, for example, we released the model itself, the data that was trained on, The code, both for training and inference all the logs that we got through the training run, as well as ev…”
Soldani: Ai2's OLMoE is state-of-the-art in its size class
“So a few things that we released this year was, as I was mentioning, this OMOE model which is, I think still is state-of-the-art MOE model in its size class.”
Soldani: OLMo 2 is the best state-of-the-art fully open language model
“And finally, the last thing we released this year was Olmo II which so far is the best state-of-the-art fully open language model.”
Soldani: Content owners blanket block crawling due to closed AI models
“What they found is, as a reaction to, like, the close like, of the existence of closed models, like OpenAI or Cloud GPT or Cloud a lot of content owners have blanket blocked any type of crawling to their website.”
Soldani: Web blocking disproportionately benefits incumbent closed AI labs
“And I think the problem is this blocking or ideas really, it impacts people in different ways. It disproportionately helps companies that have a head start, which are usually the closed labs, and it hurts incoming newcomer players where you either have now to …”
Soldani: AI Risks Are Standard Software Issues, Not Existential Threats
“The thing that it's like to me is sorry, it's ingenuous, is like just putting this AI on a pedestal and calling it like an unknown alien technology that has like new and undiscovered potentials to destroy humanity. When in reality, all the dangers I think are …”
Soldani: Open Model Bio-Risk Warnings Were a Lobbying Ploy
“You know, if you remember the beginning of this year, it was all about bio-risk of these open models. The whole thing fizzled out because there's been, finally there's been, like, rigorous research, not just this paper from coherent folks, but there's been rig…”
Soldani: Open Source AI "Dodged a Bullet" on California's SB 1047
“I look at things like the SBE, 1047 from California, and I think we kind of dodged a bullet bullet on, on this legislation. We, you know, the open source community, a lot of the community came together at the last, sort of the last minute and did a very good e…”
Soldani: Companies release open models for commercial interest, not long-term commitment
“I think there's a lot of investments in companies that at the moment are releasing their model in the open, which is really cool. But it's usually more because of commercial interest and not wanting to support this like open models in the longterm.”