Prediction Not checkable as stated
Mensch: AI export controls will not stop European or Asian progress
“Effectively, if there's some export control over weight this is not going to stop any country in Europe, any country in Asia to continue its progress. And they will collaborate to actually accelerate that progress.”
Assertion Supported
Mensch: Mixtral matches Llama 2 70B performance at one-sixth the cost
“Mixtral is actually on par with Lama-to-seven TB while being approximately six times cheaper or six times faster for the same price.”
Opinion
Mensch: AI regulation should target applications, not foundational math
“What you want to regulate is the application, and the issue we had, and the issue we're still having now is, We hear a lot of people saying we should regulate the tech, so we should regulate the function, the mathematics behind it, but really you never use a l…”
Opinion
Mensch: FLOP count is the wrong metric for regulating AI models
“Pre-market conditions like flops, the number of flops that you do to create a model is definitely not the right way of doing of measuring the performance of a model.”
Prediction Not checkable as stated
Mensch: AI will impact every country's GDP by double digits
“It will have an impact on GDP of every country in the double digits in the coming years.”
Prediction Not checkable as stated
Mensch: General-purpose base AI models will eventually become open source
“There are general purpose models like base models, compression of the web. That are eventually going to be open source and that can serve as the right basis for constructing specialized systems.”
Insight
Mensch: Nations and enterprises must own their AI cultural alignment
“For me, this is an inherent limitation of centralized AI models, where you're thinking that you can encode some universal values and some universal expertise into a general purpose model. At some point you need to take the general purpose model and ask a speci…”
Assertion Partly supported
Mensch: Mistral Sabah outperforms Arabic AI models five times its size
“Today our model it's the 24 B. It's called Mistral Sabah. It's a model tune in Arabic. Is outperforming every other language model that are, like, five times larger.”
Insight
Mensch: Closed-source AI models are unfit for high-certainty applications
“You can evaluate a model much better if you have access to the weights than if you only have access to APIs. And so if you want to build certainty around The fact that your system is going to be a hundred percent accurate, I don't think you should be using a c…”
Assertion Supported
Mensch: 2020-2021 AI research suffered from flawed scaling laws in GPT-3 and Gopher
“There was also a misconception on GPT-free and basically in 20, 21, every paper made this mistake.”
Insight
Mensch: Compute-optimal LLM scaling requires equal relative growth in parameters and data
“In common words, if you multiply by four your compute capacity, you should multiply by two, the model size and by two, the data size.”
Insight
Mensch: Pre-trained AI models should be neutral without creator bias
“Pre-trained models should be neutral, and we should empower our customers to take these models and just put their editorial approaches, their instruction, their constitution, if you want to talk like entropy, Into the model. So that's the way we approach the t…”
Assertion Not checkable as stated
Mensch: Internal Mistral models rank among top three globally
“Internally We have stronger models that are in between 3.5 and four that are basically second or third, the second or third best model in the world.”
Assertion Not checkable as stated
Mensch: Open-source AI trails proprietary models by six months
“So really we think that the gap is closing. The gap is approximately six months at that point.”
Prediction Not checkable as stated
Mensch: Open-source models will equal proprietary AI performance
“But I really think that will converge to a setting where you have proprietary models and the open source models are just as good.”
Assertion Not checkable as stated
Mensch: Fine-tuning access makes GPT-4 easy to exploit into bad behavior
“It's actually super easy to exploit an API. It's super easy, especially if you have fine tuning access to make GPT-IV behave in a very bad way.”
Insight
Mensch: AI companies must run on distinct product and scientific frequencies
“You have fast frequencies on the product side. It's iterating every week. And you have slow frequencies on the science side that are looking at why profoundly the product is failing on certain domains and how they could fix it through research, through new dat…”
Insight
Mensch: Asynchronous AI workloads will drive massive new compute demand
“We are moving to, towards workloads that are more and more asynchronous. So workloads where you give a task to an AI system, and then you wait for it to do, like, 20 minutes of research before returning. So that's definitely changing a bit the way you should b…”
Insight
Mensch: Overtraining models past Chinchilla limits lowers inference costs
“If you take into account the fact that your model Should also be efficient at inference time. You probably want to go far beyond the Cinchilla scaling low. So it means you want to overtrain the model. So train on more tokens than would be optimal for performan…”
Insight
Mensch: Mixture of experts decouples model capacity from inference cost
“A sparse mixture of experts, you take the dense layer and you duplicate it several times. And so that's where you actually increase the number of parameters. So you increase the capacity of the model without increasing the cost. So that's the way of decoupling…”
Prediction Held up
Mistral AI plans to monetize through an open-core business model
“As a business, we do need to have a valid monetization approach at some point. But we've seen many businesses build open core approaches and have, A very strong open source community, and also a very good offer of services, and that's what we want to build.”
Assertion Supported
Mensch: Mixtral matches GPT-3.5 performance
“So mixed trial is as similar performance to GPT, 3.5.”
Assertion Not checkable as stated
Mensch: Fine-tuned Mistral 7B matches GPT performance on specific enterprise tasks
“They took Mistral-Seven-B, had a lot of human annotations, had a lot of proprietary data, just modify Mistral-Seven-B so that it solved their task, just as well as GPT-PT-PT-PT, but only for a lower cost and a higher level of control.”
Assertion Supported
Mensch: Hugging Face DPO outperformed Mistral AI's initial instruct model
“The hugging face folks first did the direct preference optimization on top of Mistral seven B and made a very strong, a much stronger model than the instructive model we proposed at the early release.”
Arthur Mensch