2022 DeepMind paper showed dataset size matters more than parameter count
Anjney Midha · Safety in Numbers: Keeping AI Open · Dec 28, 2023 · at 0:12
A voiceover narrator introduces Arthur Mensch's research background, highlighting his co-authorship of DeepMind's influential 2022 Chinchilla paper on compute-optimal scaling laws.
0:00 / 0:13exact quote · 13.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“In fact, in twenty-twenty-two, a pivotal paper came out that changed the way that many people in the research community thought about this very calculus. And it demonstrated that datasets were actually more important than just the sheer size of the model.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Anjney Midha
Midha: Slowing frontier AI delays critical medical breakthroughs like curing cancer
“Until we've solved cancer, every month that we're not rushing to the frontier of accelerating biological discovery or scientific progress is a month that millions of people are suffering from disease that we could be solving with the eye.”
Midha: Existing regulatory tools are sufficient to manage AI risk
“Marginal risk is, I think, a slightly just more technical way to say, we have the tools to manage risk. We don't need new ones.”
Midha: Claims of a multi-year US AI lead over China were false
“No empirical evidence at the time that this was, that AI was creating net new marginal risks, and just factual inaccuracies that we were ahead of China, because if you just paid attention to what's happening, DeepSeek had already started to publish a fantastic…”
Midha: Distilling frontier capabilities from US AI outputs is easy
“No, actually it's not that hard to distill on the outputs of our labs.”
Midha: Downstream liability for open-source AI developers destroys the innovation ecosystem
“The status quo is you do not hold scientists liable for downstream uses of their technology. That's absurd. That's a great way to shut down the entire innovation ecosystem and start throwing literally like researchers in jail.”