Mensch: 2020 OpenAI scaling laws paper was poorly executed
Arthur Mensch · No Priors Ep. 40 | With Arthur Mensch, CEO Mistral AI · Nov 9, 2023 · at 4:36
Arthur Mensch, CEO of Mistral AI and co-author of DeepMind's Chinchilla paper, critiques the earlier 2020 Kaplan et al. scaling laws paper.
0:00 / 0:12exact quote · 12.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Basically the story was that everybody was training models on too few tokens because of the paper from 20 20 that happened to be not very well executed.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Arthur Mensch
Mensch: AI base models should know everything and use modular guardrails
“Assuming that the model should be well behaved is, I think, a wrong assumption. You need to make the assumption that the model should know everything. And then on top of that, have some modules that moderate and guardrail the model.”
Mensch: AI safety should regulate application makers, not trust big model labs
“And the way you make this healthy competition is not By trusting a couple of companies to do their own safety, it's rather for, it's rather the way you do it is to ask application makers to comply with some rules.”
Mensch: LLMs are not marginally more dangerous than web search
“Nothing is showing that LLM is actually marginally better than a search engine to find knowledge on, on topics that would enable bad use.”
Mensch: Banning open-source AI enforces regulatory capture for incumbents
“Today going, banning open source, preventing it from happening is really a way, well, to enforce regulatory capture, even though the actors that would benefit from it, Don't want it to happen. But by design, if you actually ban small actors from doing things i…”
Mensch: AI bioweapon risk narrative relies on unscientific circular citations
“No scientific studies in proper form was published, but then policy papers started to cite Non-scientific papers arguing that these were scientific evidences that the bioweapon narrative was actually true. And then policy papers started to cite the other polic…”