The Exchanges

Q to be used by billions of people. So what's the updated state? Of, of scaling laws. I think there was obviously the Kepler, and then there was Chinchilla, and then people kind of called the LAMA scaling law, like the 100 to 200 X kind of parameter to token ratio. What's your updated thinking on how to think about scaling laws when you pick model size and, and training data?

A Right. So, you know, as you said, this Kaplan paper with scaling laws, but they figured out, basically they tried two dimensions. The model weights, and the number of training time, like number of steps, training tokens, epochs. And for that, they figured that model size is what matters, so GPT-free was way too big compared to the actual number of training tokens, because they did a mistake not adapting the scheduler. That's what Chinchilla emphasized and discovered. To be fair, I think OpenAI knew that at the time of Chinchilla paper. But yeah, basically, Chinchilla said, We have to revisit the scanning laws originally published by Kaplan and emphasize much more the importance of training tokens. And they did like some really good scanning laws showing that there's an optimal, basically you need to double the number of training tokens every time you double the training weights to get an optimal ratio so that for a finite number of compute, you will end with the best results in your paper. And what I call the Chinchilla trap is that that's good if you want the best flagship model that obtains the highest performance on your paper. But if you want to use your model at inference time, inference, the two dimensions, one remains the model weights, but one drops the number of tokens you train it, number of steps. And so, to be compute efficient at inference time, it's much better to…

AI assessment note: “if you consider inference time, you need to go beyond Chinchilla.”

Q in the vocab, you have this kind of like concepts representatives token. So usually the larger the vocab size, the more nuanced the model can be about thinking about different things. What are the scaling laws of those organizers? You know, is a 120 a K kind of like very large and it doesn't really matter. Like, do you want to double it? Like, any thoughts there would be great.

A There's a lot of dimensions to take into account here, I think. The first thing obvious to say is Lama III compared to Lama II is multilingual, has multilingual capabilities. We worked on that. And so, because you have languages that are not just Latin languages like English, there's a lot of different characters, uh, you want to include them, uh, to represent, like, special words there, and so you need to have a bigger, uh, vocabulary size. That's the obvious thing. Which is also probably why GPT-Force Zero Uh, has a much bigger vocabulary, as it's like naturally multilingual, multimodal, uh, and speech. So, that's why we went to from 30 to 128 vocabulary size. The interesting thing, I think, to discuss about tokenizer is about scaling laws related to that. If you increase your, uh, vocab size, well, you have a bigger matrix, Which takes longer to compute. It depends on the model size. But for a small model, it has a much bigger impact than a bigger model. So, increasing that, basically saying otherwise, the number of vocabulary size, 428, is the same than the eight, 70, or four zero five b. But so, relatively in percentage of the total number of weights, for the seven b, it's much more than the four zero five b. Well, it's small compared to total number of weights. So, that has more impact in terms of training speed. There. But what is interesting is, with a bigger vocabulary…

AI assessment note: “The interesting thing, I think, to discuss about tokenizer is about scaling laws related to that.”

Q so many different usage of this thing, but specifically synthetic data for pre-training, it feels almost like you're running multiple epochs On the raw data when it's, while it's rephrased or reformatted by a language model, right? And in my mind, it's very similar to computer vision, where you do data augmentation on an item, right? Like we're, we're doing data augmentation. That's the less cool name for synthetic data.

A That's very interesting. I totally agree with you related to pre-training, totally a standpoint you said. I think it's very different though for post-training in the future direction on synthetic data that I'm personally excited. Like, for instance, what I'm excited about is we had this survey on augmented LLM a year ago, and all the idea is, like, if you augment your LLM with something else, it can be a retriever, it can be search, it can be a tool, it can be a calculator, it can be a code execution. Then you are not just distillating, like, doing some data augmentation with your model, But you're actually adding some expert skills that possibly goes beyond the model weights. For instance, if your model, like, can calculate something it was wrong before, and now it has access to a calculator, and you can retrain your model on that, then you're learning something new. If your model didn't know something about LAMA two probably doesn't know a lot about LAMA three, but now if it can search online about it, and then you train the model on that, Then you have a positive feedback loop, like what we call expert direction, targeting directly the weakness of the model. It's like continual augmentation of the language model, much beyond just that augmentation.

AI assessment note: “I totally agree with you related to pre-training, totally a standpoint you said.”

Q physicians are rated not empathetic or slightly empathetic versus all the model responses are rated very empathetic and empathetic at worst. You know, most people might look at it and not really get much from it, but obviously it resonated with you. Can you run people through like some of the choices you make in post training to like optimize for one of the two and getting the best responses?

A I think the tweet was about like, The intuition of why reinforcement learning with human feedback works. When we started, uh, Lama Two, I had, like, this budget of annotations in millions of dollars, and, okay, what to do? I'm responsible of that. I'm accountable for a model at the end that can follow instructions and compete with GPT-A-V at the time. What to do? You can annotate supervised venturing data, which refers to a human to create a prompt, and to also write itself, himself, the answer expected by the model. So then you train on that, and in a supervised manner, uh, that's like very classic and standard on Fine-tuning, machine learning. The other thing is reinforcement learning with human feedback, where the annotators type a prompt, but this time you sample two different answers from your model, and you ask the annotator which one he prefers, and then you will train on the preference, basically, to simplify. When you ask to train on the preference of the model, that seems very weird, and, uh, not really robust. Training on synthetic model generated by the model. So I was like, let's annotate 100,000 more of supervised venturing data, and let's annotate a bit of preference to do RLHF, because everyone is doing it. And we had this human evaluation, after a few weeks in LAMETO projects, where our model was already better than the annotation from the humans. So, you get a…

AI assessment note: “You can annotate supervised venturing data... The other thing is reinforcement learning with human feedback”

Q really liked it. In Lama Tree, you dropped like the intermediate weight. So it's a eight, 70 and now 405 B. What was the thinking behind going so large? I mean, you talked about the hardware capabilities at inference, like I cannot run a 405 B model at home for sure. And it might be hard to even get the cloud resources to do it. What was the decision there?

A The decision is super simple. We want the best model. We want to be number one and number two. We started one year and a half ago, and we did quite some journey. We filled the gap with GPT-IV, so that will be the first open source model that actually compares to GPT-IV. There's now GPT-IV-Zero, of course, and we're closed, but we're not there yet. Not in all capabilities, but the gap is getting smaller and smaller. There's also, like, what compute we had at the time when we started the run in January. We put a lot of effort there, but as, like, Mark announced, we have more and more GPUs, so the next generation will be bigger. So that's what drives the decision. Now, maybe let me reflect two things he said. You cannot use it at home. That's probably true. But quantizing it to FPA can run on node, even with a long context of 128 tokens. Second thing is, I'm hopeful that the community will lead to a lot of findings by open sourcing it, And there is smart way to actually make you use it on your computer. If you remember number one, number two, like when we published models, people were saying it's too big, and after two weeks, it was running on a Raspberry. I don't know if it would be the same, but I hope the same kind of trend. And by releasing those models, we are enabling that. Now, the last thing I want to add is having bigger models enables to collect better data, for instance…

AI assessment note: “The decision is super simple. We want the best model.”

Q Okay. You talked about agents. So I guess we should probably mention the, the work on agent stuff. And you also, you know, pre-conversation, uh, mentioned that you're already starting work on Lama IV. What does agents have to do with Lama IV? How does your work on Gaia, you know, inform all this work?

A Yeah, you know, so we published one year ago Gaia, a general assistant benchmark, that followed a direction I really like pursuing. I mean, everyone passionate about AI and trying to build Jarvis will go there. So I did Toolformer and this survey on augmented models. In fact, you know, reflecting back, I was, okay, we have Galactica, we have Lama One, we have Toolformer, And there's, like, GPT-FREE.FIVE at the time, and then four. If you don't have a good instruct model to follow instructions, the extension and the future of toolformer is limited. So we need to work on that, and we did LAMA-II, and then now LAMA-III. And it's very interesting. On General Assistant Benchmark, so Gaia, agents powered by language models perform to zero with GPT-FREE.FIVE, and to something very, uh, significant, like 30, 40%, 60% with GPT-FOUR. So there's a gap of intelligence here, and I think this gap of intelligence, this threshold that you pass, in terms of zero-shot function calling, following complex instructions that can span over a page of constraints, all those things that makes the current, the nowadays agents, with the react loops, pre-planning, multi-steps reasoning, function calling, work in practice, is like this gap of intelligence. So now that we have Lama-free, I'll be back to agents. I expect some incremental and significant progress on pre-training and post-training, but I'm real…

AI assessment note: “So now that we have Lama-free, I'll be back to agents.”

Q Do you have an intuition on which areas work better for which? For example, you mentioned the physicians are experts. What about maybe, like, code or, yeah, you also have a multi-model working on, so, like, image generation is like, or does this apply to any modality, any subject?

A That's an open research question. The intuition in general is that, like, for instance, for code, because this is factual, You can check if the code is correct or not. RLHF is not the way to go. You prefer to do, like, supervised fan tuning as a human to write a code. But in fact, because humans make mistakes, because actually, even in code, there are some preferences that emerge, things like that, and maybe for some other reasons that we don't know, RLHF is so much more scalable, it goes less, it's easier, that it leads, in general, to just better performance. And maybe we can come with a compromise. We actually suggested teacher forcing. In Lama-free. A new method that kind of fills the gap between, not teacher-forcing, sorry, teacher-critic. Teacher-forcing is going to train the models. Teacher-critic where it reconciliates and unifies super-assend tuning and LHF, such that when you do human preference, and you have two outputs, but both are very bad in the code, for instance, you will ask the human to edit the best answer to make it correct now. So now you are doing SFT when all the answer was really bad, So that you can get out from the local minimum.

AI assessment note: “That's an open research question. The intuition in general is that, like, for instance, for code”

Q We interviewed him at iClear. Both are agent startups. What do you think is like the direction that startups should be working on, you know, agent-wise and maybe what, what is not working?

A That's a tough question. One thing I say quite often is deep learning has these very specificities that makes it challenging to predict that it's self-destructor. Self-destructive technology. Since that, think like, you know, Grammarly, this technology like where the startup you plug play and it corrects your grammatical errors. Everyone told them, guys, deep learning, create a barrier to entrance, annotate data, create data. And they had a lot of data for that, and the next day, with the same exact technology, deep learning, someone comes with ChatGPT and tell them, yeah, I can do the same, better, and so many other things. Zero barrier to entry, they, from, Yesterday to today. And what is crazy here is that it's based on the same technology. And so there's a lot of people working nowadays to try to mitigate issues with current generation of models. And I'm telling them, like, assume always the next generation will get better. So if your business will benefit from a new generation with better abilities, that's a good business. If your business may be replaceable, and if all the work you have done May vanish and be, like, wasted because there's better models, then maybe change.

AI assessment note: “if your business will benefit from a new generation with better abilities, that's a good business”

Q went into LAMA-III, I think there's a lot of chatter obviously about synthetic data and like, uh, there was the rephrase the web paper that came out maybe a few months ago about using, you know, Mastral to make training data better. Any learnings from that? It's like, is there, how much can you rewrite with the models? Like, uh, I'm sure people would love to hear more about it.

A Right. So it's very interesting, the research direction, synthetic data in general, synthetic data for pre-training. My intuition is that the web It's full of shit in terms of text, and training on these tokens is a waste of compute. Just having a good classifier that labelized that is cool, and LAMA was the best, at the time, before LAMA Free, the best model we had access to, uh, legally, to labelize the web and select what are the good tokens and the bad tokens. The additional thing is that it also enabled to have a topic tag, Like, is it about law? Is it about politics? Is it about chemistry, math, reasoning? So that you can also adapt a bit the mixture to like balance a bit more the diversity.

AI assessment note: “the best model we had access to... to labelize the web and select”

Q from English. When you learn English, you learn language and the language just translates to other forms of languages, especially if they're the same family, right? Like, uh, yeah, so. Maybe we should get right into Lama two, spend a little bit of time there, and then, and then we'll go into Lama three. So like, uh, what is the story of Lama two from your point of view? Yeah.

A So as I was saying, I started to Meta on Galactica. That was one of the first large language models at Meta, the language model for science. We released it in, I think December or end of November, I don't remember, one year and a half ago. I don't know if people remember, but it was huge on Twitter. Both with people, like, thinking it's the end of science, and, like, that will be a lot of hallucinations in papers, although, as well, like, it's super awesome. I still think it was super awesome, but, you know, we didn't do, like, instruction tuning, Garelechev techniques at the time. It was a weird moment, because two weeks later, ChatGPT came out, and that's the moment where, like, I think all the fan companies went upside down, and where we had a huge traction from leads, To now work on that and make a ChatGPT as soon as possible. So we had this one, two months of, like, what to do. I actually was working on Galactica Instruct, which basically you could connect it, we had a partner with Overleaf, the Google Doc of, like, scientists, where you can write papers, and it's in, you write there in LaTeX, you have to do a lot of citations, so the idea was that you can just, like, ChatGPT or GPT Instruct, Ask, uh, or swap two columns in a LaTeX table. That's something very, very, uh, time-consuming, I can promise. You could, like, say, oh, find me a citation about LLMs and bias. We'll …

AI assessment note: “So as I was saying, I started to Meta on Galactica.”

Q B, 33 B, 65 B model sizes. And then Lama two, seven, 13, 70. How do you kind of evaluate what's worth training? Especially when you think about data, it's like, you know, maybe a 100,000 is enough for like a seven B model, but it's not enough for a 70 B model. How do you decide model size, especially when you're maybe annotation constrained on some of these things?

A That's a very good question, and there's no good answer. There's so many parameters to take into account from the scaling loss at training time to get the best performance. The GPU constraint, and on what different hardware, and we think about meta, but also of the community, and like people are not just using H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H All those constraints makes it very, very challenging. At inference time, you have a lot of costs. So how to trade off between inference costs and training costs? It's a very challenging problem. In general, we tend to think in particular for Lama three. Lama two, maybe I would say it's like Lama one. We had a flagship model, which was seven TB. It's also because the project was taking some routes to reproducing a chinchilla, which was a seven TB. For Lama three, we also like move to one size more. The flagship model for the O five B. I think that there was also the question of, we want a model at this time. We have this amount of compute. Given the scaling levels and the amount of tokens, we have to train it. What would be the right balance to still like fits in a different style? So we try to, to have some trade-off like that.

AI assessment note: “So how to trade off between inference costs and training costs?”

Q Interesting. What are areas that you had to work the hardest to catch up to like the private models? Maybe like there's, you know, not as good public data or whatnot, or is performance improvement just kind of even across the spectrum?

A Honestly, all of them, we are behind all of them with, uh, between LAMI-II and, uh, GPT-IV. I mean, It's different challenges every time. Like, being good at code or reasoning is something we didn't do at Lama too, so we had to build everything from scratch. Improving on helpfulness, and, which is one of the main dimensions that people look at, I think, in Zarena. But, which is, by the way, a very interesting evaluation, because when we did the preview, and I don't know yet what would be the results for this new Lama-free, but we ended very high in this blind test leaderboard, and to be honest, I didn't expect that. I knew we had good results internally, but how that will transfer to perception from the community, people, like, using it in practice and comparing it to their others, I didn't expect that positive feedback. That high ELOS score on this benchmark. It doesn't say, like, everything. As I said before, which is also interesting, because it's the community that judge the prompts, and create the prompts, and judge the answers. We are limited, we are not, like, good to do that, and so it gives you a very good indicator of how good, helpful, how on the main core of the distribution, simple prompts, about the tone of the model compared to the ELOS, but for much more complex prompts, much more intelligent, Reasoning, coding, have complex stuff. It doesn't tell the, the full …

AI assessment note: “Honestly, all of them, we are behind all of them”