Pineau: Meta's 2019 RoBERTa model hit 20M monthly downloads during LLM hype
Joelle Pineau · Cohere's Chief AI Officer, Joelle Pineau: Why Scaling Laws Will Continue & Future of Synthetic Data · 20VC with Harry Stebbings · Nov 3, 2025 · at 57:31
Joelle Pineau, Chief AI Officer at Cohere and former Meta AI research executive, highlights strong developer demand for smaller, practical AI models.
0:00 / 0:13exact quote · 13.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We were in, in the frenzy of large language models, and I pulled the stats on, you know, most downloaded models of last month. We had a model like Roberta from 2019, small language model, was getting twenty million downloads a month.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Joelle Pineau
Pineau dismisses catastrophic AI existential risk predictions as unscientific
“I don't have a lot of patience as a scientist for people who are predicting the extremist scenarios, the catastrophic risks of AI, you know, AI becomes our overlord kind of scenario.”
Pineau: Out-of-the-box reinforcement learning will not deliver AGI
“Now, you know, where we're maybe getting a little bit ahead is thinking that just RL out of the box is gonna give us AGI. That part, a lot less so. You know, if you look at the curve of progress, RL is terribly inefficient, and so the amount of signal you need…”
Pineau: Reinforcement learning is fundamental to AI and will not disappear
“Oh, I'm still super bullish on RL in that, like, the concept itself is so fundamental. You know, this idea of training through a system of rewards, of indicating what's valuable and what's not valuable through numerical values, like, that is so fundamental. It…”
Pineau: Using reinforcement learning to teach AI social behavior remains unsolved
“RL, to shape the behavior of models, to get them to be social creatures, that we have no idea how to do. I mean, I don't know if you have children, but like shaping their behaviors, you know, the number of times you can repeat the same thing, and still they do…”
Pineau: Compute and Data Scale AI Linearly, While Algorithms Cause Step-Functions
“Compute and data have a more linear effect on progress. You build more compute. You run bigger models. You can typically get better performance. You feed in more data. It's not just quantity. You need to worry about quality and diversity as well, but roughly i…”
Pineau warns against betting that AI compute scaling will hit a wall
“They've been, the scaling laws have been remarkably robust. They don't play exactly as we expect, but still, they've been remarkably robust. Lots of people have bet against scaling laws in the past. And I would say overall, you know, we've seen a pretty, prett…”