Mensch: Mixture of experts decouples model capacity from inference cost
Arthur Mensch · Safety in Numbers: Keeping AI Open · Dec 28, 2023 · at 10:59
Arthur Mensch, co-founder of Mistral AI and former DeepMind researcher, explains the fundamental architectural advantage of Sparse Mixture of Experts models.
0:00 / 0:17exact quote · 17.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“A sparse mixture of experts, you take the dense layer and you duplicate it several times. And so that's where you actually increase the number of parameters. So you increase the capacity of the model without increasing the cost. So that's the way of decoupling the memorization and what you can remember the capacity of the network to its cost at inference time.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Arthur Mensch
Mensch: AI export controls will not stop European or Asian progress
“Effectively, if there's some export control over weight this is not going to stop any country in Europe, any country in Asia to continue its progress. And they will collaborate to actually accelerate that progress.”
Mensch: Mixtral matches Llama 2 70B performance at one-sixth the cost
“Mixtral is actually on par with Lama-to-seven TB while being approximately six times cheaper or six times faster for the same price.”
Mensch: AI regulation should target applications, not foundational math
“What you want to regulate is the application, and the issue we had, and the issue we're still having now is, We hear a lot of people saying we should regulate the tech, so we should regulate the function, the mathematics behind it, but really you never use a l…”
Mensch: FLOP count is the wrong metric for regulating AI models
“Pre-market conditions like flops, the number of flops that you do to create a model is definitely not the right way of doing of measuring the performance of a model.”