The Exchanges

Q Yeah. Yeah. Uh, cool. Uh, do you, uh, just a side follow-up, uh, did you actually follow the sort of, uh, schedule free optimizer stuff from last year with Aaron Defazio? Anything came out of that?

A I don't have a strong opinion. I tried it, uh, like a few days after it was released because I was, uh, we were writing a paper about the WSD schedule. And so this was kind of an alternative to it. And, uh, it didn't perform as well as like, uh, Simple Adam W and it was more sensible to some, to the beta one and beta two. But I think like with this kind of stuff, you, you really need to have this, uh, this knowledge on how to optimize, uh, those hyperparameters for each optimizer to really get the best performance of each one. And, uh, so I don't want to, to say something bad about it because basically at 19%, uh, there is a 90% chance that it's just me that used the wrong hyperparameters.

AI assessment note: “I tried it, uh, like a few days after it was released”

Q And for these five steps, how do you think about which ones have the highest variability? I feel like some of them, there's kind of like a meta of this is how you get the best results. What are like the parts that are like still very not figured out where like you feel like the most interesting research has been done?

A I think data sets is like the data is like It's a bit unexplored because, uh, I think people don't find it very interesting. I mean, for, for a bit of time, like people were a bit under exploring like the data sets, especially since it's super important. Uh, but we, we've seen like recent paper and recent trend to build better and better data set. And so I'm, I'm confident that like, uh, hopefully people in the open will Somewhat catch up with, like, the, the, the big models. The rest of it, I think the model architecture, we, with, like, the Quen, Deepsea, Kimi architecture, we're already at a point where this is already not optimal, but this is, like, pretty, pretty advanced. And there is all those, all these things where basically you want to, to maximize, uh, like to, to, to do some new optimizer and to, to really, uh, have some good stability and scaling feature. This is a bit on the explore and especially like since Quinn and Dipsy paper doesn't really detail or they do the stuff. I mean, we've left with a very few, uh, model provider, even in the open that, Talk about all of this. And for example, a good, uh, a good way to view that is that, uh, DeepSeq rig three is still using the same Adam parameter than, uh, Lama two. I mean, there is something wrong with it. I don't know if it's just me, but I feel that like the, the, the, the hyperpenters for Lama two AB, for exampl…

AI assessment note: “I think data sets is like the data is like It's a bit unexplored”

Q Awesome. Um, yeah. Okay. Uh, anything else on the optimizer world that's going on? It's like, uh, Interesting to you.

A Yeah, there is this, uh, this part from the, the Asian Sol AI paper about Muon, and the, the idea of this plot is basically saying that you can use MuP to, I mean, you can use some technique to, to do hyperparameter sweep, and it's working with Muon, for instance, and there is this idea where you, so in this, in this paper, they only did, uh, one D, uh, Hyperparameter sweep. So they fix everything and they just change the learning rate, for example. But for instance, there is some connection between the learning rate and the weight decay or learning rate and batch size. But you always want to sweep together some of the hyperparameters. And this is what they, they, they do here. For example, they, they, they sweep both the, the learning rates and the weight decay together to, to find the optimal one for, for the training.

AI assessment note: “there is this, uh, this part from the, the Asian Sol AI paper about Muon”

Q How easy is it to divide the skills? So when you say, you know, literature, math and code, what can you maybe run people through the process of like, how do you kind of like funnel these things and different experts? And then is it just as easy as you said to like, I'll just delete the letter literature part, or is there some sort of like codependence between them?

A Yeah, there is some codependence definitely. So there is other way to, Basically limit this codependence, but there is definitely some codependence. And when you do printing, I, I don't know if people are doing that. I think that's interesting to explore, but, uh, like at least in the open, people are not really like printing, for example, the English literature. That was just some, some research idea that could be called in the future. But, uh, but yeah, but the, the specialization is actually done in the router layer of your MOE. And this is pretty important basically because this is like the, the, the step where you will learn, like each token will learn to which, uh, experts to go depending on like the, the, the meaning of each expert. And, uh, and we can even have some fancy stuff. So this is like, uh, from two weeks ago, this long cut flash from the, the DoorDash of China paper that released the company that basically released like a state of the art model, which is pretty crazy.

AI assessment note: “there is some codependence definitely... specialization is actually done in the router layer”

Q literature. Do you see a future in which the MOE model is like, you know, Front end expert, like, you know, if it's like a TypeScript code base, it's like, oh, this expert, like the routes is an expert at, you know, the component is an expert at the test writing. Like, is that something that makes sense? Are we bottlenecked by the data or what are people doing there?

A So my intuition, I, I, I don't really know, but my intuition is that, for instance, between math and, I don't know, all the history, uh, like the Greek history, for instance, not really the Greek because it's related to math, but let's say code and Greek history. There is not a lot in common, except just the fact that the, the language is the same. But between like, for example, I don't know, TypeScript and JavaScript or Python, and I don't know what language, there is more similarity. So I'm not sure, uh, if it's, like, there is, there is maybe more transfer learning, meaning that a model train on, like, Python and JavaScript will benefit from the JavaScript data as well. I'm not sure it's looking for Python or Greek history.

AI assessment note: “there is maybe more transfer learning, meaning that a model train on”

Q excitement about open models, local models, blah, blah, blah. And then I think, you know, maybe the reality is that there's not as much adoption as people would like. Is there anything You wish people would do more of maybe like, uh, you know, underrated use cases for which you think the smaller models are like just as good as the frontier ones that people are not taking advantage of?

A Yeah, I think even I, to be honest, I'm mainly using a closed model because it's easier, basically. It's like the same interface when I'm like, for example, researching very deep on papers. And when I'm, for example, doing like translation stuff, I want to translate something from English to French. But I know some of the easier, uh, user are using this one for translation. There is also obviously all the on-device application, which are super important. Uh, there was this Apple keynote two hours ago where they, they, they, they have said the word AI a thousand times probably. And yeah, so obviously on-device, uh, applications are super important. And, and yeah, for the future of SmallLM, I think our goal is basically to build the model that The community don't have and need. So we are super open to feedback. Uh, don't hesitate to, to tag me on X or any face or whatever. If you, if you have like a, um, if you think that there is like some gap in the, the, like what the, all the open source models are today, and you, you wish they were smaller, for example, very small model, like under one billion parameters or something like that.

AI assessment note: “translation stuff, I want to translate something from English to French... on-device application”