Oct 13, 2024 · 1h 12m · latent-space
[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz
Vibhu Sapra · 33m spoken Nathan Lambert · 4m spoken Eugene Xia · 2m spoken Demo User 2 · 4s spoken Demo User 1 · 2s spoken
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this Paper Club session, speakers Vibhu Sapra, Nathan Lambert, and Amgadoz analyze the architecture and data curation pipeline behind AI2's open vision-language model family Molmo and Pixmo, followed by a technical deep-dive into the pruning and performance of OpenAI's Whisper Large v3 Turbo.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 0.1 Guest teaching 5.1 Guest disagreement 1.2 The hosts pushing back 0.1
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 12:00 Nathan demystifies vision model breakthroughs
Nathan downplays the novelty of recent vision-language advances, arguing the field is simply underdeveloped and that frontier labs will easily ingest their fine-tuning methodology.
Hardest push from the hosts ▶ 29:04 Vibhu questions benchmark overfitting on clocksVibhu pushes back on the inclusion of synthetic clock datasets, questioning whether it represents genuine capability generalization or merely overfitting to a narrow benchmark.
Biggest teaching moment ▶ 1:05:01 Amgadoz explains Whisper 30s padding requirementAmgadoz educates the group on Whisper's architectural constraint requiring 30 seconds of audio padding even for 300 millisecond streaming chunks.
The host holds their own ▶ 25:21 Vibhu clarifies synthetic QA pipeline without vision LLMVibhu clearly explains the synthetic bootstrapping mechanism where text-only LLMs generate QA pairs from dense OCR and speech captions to train multimodal models.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Nathan Lambert on Vision Space Data Strategy | 1 | 6 | 3 | 1 | Nathan Lambert demystifies the hype around Molmo, stating the vision space is heavily underdeveloped and proprietary frontier labs will simply replicate and absorb the open fine-tuning data. He also reveals internal AI2 timeline pressures and model design trade-offs regarding chat templates. | |
| Pixmo Speech-Based Data Collection Pipeline | 0 | 4 | 1 | 0 | Vibhu outlines Pixmo's speech-based data generation pipeline using Whisper transcription and LLM normalization. Nathan briefly corrects Vibhu regarding the distinction between MMLU and the multimodal benchmark MMMU. | |
| Pixmo Subsets: Pointing, Clocks, and QA Generation | 0 | 6 | 2 | 0 | Eugene and Vibhu explore the Pixmo QA generation and clock datasets. Nathan explains the pragmatic behind-the-scenes reality: the lead researcher became fixated on fixing clock reading, yet the resulting model still fails to generalize to simple dials. | |
| Molmo Benchmark Performance and Human ELO Evals | 0 | 5 | 2 | 0 | The panel assesses Molmo's benchmark performance and human ELO evals against proprietary models. Nathan discusses how using OpenAI's CLIP vision encoder stretches the formal definition of open-source AI and why Molmo skipped RLHF. | |
| Community Q&A on Caption Granularity and Generalization | 0 | 3 | 1 | 0 | Audience member Sam asks about required image caption granularity. Nathan acknowledges having no definitive answer, while Eugene offers the hypothesis that diverse QA pairs matter far more than granular captions to prevent overfitting. | |
| Molmo Video Demonstration and Paper Wrap-Up | 0 | 1 | 0 | 0 | Vibhu wraps up the Molmo segment by showcasing demo video clips of pointing and JSON extraction before handing over the presentation to Amgadoz for the Whisper Turbo segment. | |
| Whisper Large v3 Turbo Architecture and Pruning | 0 | 7 | 1 | 0 | Amgadoz explains the architectural optimization of Whisper Large v3 Turbo, dispelling the misconception that it uses distillation. He details how pruning the decoder from 32 layers to 4 targets the primary 90% latency bottleneck. | |
| Whisper Turbo vs. Distil-Whisper and Distillation | 0 | 7 | 1 | 0 | Amgadoz contrasts Whisper Turbo's multi-million hour multilingual pruning dataset with Distil-Whisper's English-only distillation. He breaks down the mechanics of KL divergence distribution matching versus standard cross-entropy loss. | |
| Real-Time Streaming Audio Decoding Mechanics | 0 | 7 | 0 | 0 | Amgadoz walks through the engineering details of low-latency streaming transcription, explaining how Whisper's mandatory 30-second fixed input window requires padding and small-step buffer shifts to minimize decoder execution time. |
Statements from this episode (13)
Most Open Vision Models Rely on Synthetic Data From Proprietary Models
“Most VLMs are distillations of proprietary closed source models, right? So if you need to generate synthetic data, like most open weight models rely heavily on synthetic data from private models.”
Molmo Outperforms Gemini 1.5 and Claude 3.5 Sonnet With 1M Samples
“They can get better than Gemini, 1.5, better than Claude, 3.5 sonnet, better than GPT for V at a much smaller size with about a million samples of data, which is very impressive, right?”
Speech-Based Image Annotation Produces Richer Training Data Faster Than Writing
“We ask annotators to describe the images in speech for 60 to 90 seconds, rather than asking them to write descriptions. They prompted them to describe everything in great detail, including descriptions of spatial positioning and relationships. So stuff like, y…”
Major Foundation Model Companies Will Train on AI2's Vision Data
“The things that this model is good at are things that all the foundation companies, like they're just going to take our data and train on it.”
Molmo Uses Base Model Without Instruction Tuning or Chat Template
“This is just, like, straight base model, no real instruction tuning. There's literally, like, no chat template for multi-turn. It just concatenates the messages together and, like, there's, like, go, look, good luck.”
AI2 Did Not Log Raw Audio for Pixmo Annotations
“Yeah, it was like Whisper, and I don't remember the details, and I had asked, and it's like, I don't think they actually logged the audio. I was like, oh, this would be super cool for, like, other types of multimodal, but I don't think it was in the terms of t…”
Pixmo Dataset Contains Approximately 1M Captions Across 700K Images
“They got about a million captions for 700,000 images, which I'm like, okay, that's kind of expensive.”
Molmo Reads Clocks but Fails to Generalize to Dials
“The model didn't work on clocks and then the lead was really on clocks and no models work on clocks. So they're like, we've got to make it work on clocks. One of the interesting things is that it doesn't work on dials, even though it works on clocks.”
Distilling VLMs From Proprietary Models Copies Their Spatial Pointing Failures
“For example, all this proprietary stuff sucks at clocks, so nothing that's a distillation will be good at clocks. Nothing can point if you just distill from this. If they can't point, your VLM won't point, so we show how to get good data.”
Molmo-72B Ranks Second Behind Only GPT-4o in Human Preference Elo
“Most preference ranked was GPT-Four-O, then Momo-Semety-Two-B, then Gemini, then Sonnet, then the Seven-B.”
Q&A Data Drives VLM Detail Recognition Better Than Captioning
“My intuition at least is that I suspect it's really, the heavy lifting might be actually more towards the Q and A side rather than captioning side. I view captioning as a bootstrap, but at the end of the day, we want the model to generalize what they want to d…”