Distillation

“If we do end up, you know, at some point topping out on capabilities sometime in the next few years, and distillation continues I think it's much harder to garner a premium margin if you're a frontier model company, it becomes much harder.”

“They've gotten really, really good at that, but man, all that goes away if people stop releasing these models at the frontier, and I think mythos is a side of things to come there.”

“Now, it is also the case that it's not as simple as you can take the output to these models and distill and you have exactly the model of the same capability, It's just smaller and can run fast. If that were the case, we would just do that”

“There's certainly a lot of attempts to distill models. And that comes from companies in the U S it comes from all over the world. But I think that it misses the core point, which is that the way this technology is developing is it is on an exponential. And any…”

“So there's two reasons. One is to think about distillation, but the second, in some ways more important, is that we had this insight when we first developed the reasoning paradigm that it gives us a interpretability mechanism we had not been anticipating, beca…”

“And even the Chinese models, those are a result of distillation.”

“Through distillation, which is a key technique for making the smaller models more capable, you know, you have to have the frontier model in order to then distill it into your smaller model. So it's not like an either or choice. You sort of need that in order t…”

“One of the key advantages of distillation is that you can have a much smaller model And you can have a very large you know, training data set and you can get utility out of making many passes over that data set because you're now getting the logits from the mu…”

“The more focused your use case is, the smaller you can make the model through fine-tuning or through just distillation in an even smaller architecture.”

“With distillation, you have the ability to put a number of projects into one model. It's kind of nice that you don't need to wait on all of them to complete at the same time. You can try to periodically put this together, actually make sure that as a product i…”

“Actually, distillation is like quite a big thing that people in the open are not really Really doing yet, I think”

“The capability which we have right now, maybe next year will be much cheaper to have that same thing. And that likely is the result of distillation, right? Like that's, it's not just because we are doing or figured out something magical. It is because distilla…”

“The slope of the Pareto Frontier is basically the state of the art of distillation, and the gentler the slope, the better distillation is.”

“So, so I would think maybe an optimal pipeline would look like you do distillation heavily, but then you still do some RL afterwards, because maybe there's still something you can get out of your reward functions or whatever your post-training stack is.”

“GemRTool used distillation for pre-training, where they used logits, or these soft labels, which is rather than having hard zero, one tokens, which is, I want to predict this next token, they have like soft labels for all possible tokens.”

“There's a trade-off between diversity and performance. So on the y-axis, I'm showing performance. On the x-axis, I'm showing similarity or basically how, like one minus diversity. So more similar things are less diverse. And you can see, depending on the diver…”

“The general recommendation I would give people is that maybe use a mixture of half and half. Like that's what some people have used, right? That's like saying, yeah, basically saying, I don't know what I want. I just want something to work well enough. I'll ju…”

“First step is you go to an RLHF, RLXF, whatever framework you have. You turn off the reward term. So you delete the reward part of it. All you are left is some KL term. And now you swap your KL, the anchor policy, to a bigger teacher policy. And there you go. …”

“You can actually combine RLHF and distillation together, because now you're doing two things at the same time.”

“Now, the thing is, the effectiveness of this method depends on how close the sampler, the small model is to the bigger model that we want to speed up, and actually distillation exactly fixes that, which is, by distillation, you can make things closer to each o…”

“I think the open source models, Like take Llama, for example, like they've been able to do that from their own research and perspective and data ingestion and training. And so I guess I would say distillation does not feel essential in order to unlock those th…”

“Probably the most, I would say mind blowing thing about distillation is that sometimes the smaller models become better than the bigger model through distillation.”

“I think over the last year, the big frontier labs and a lot of other people have figured out the tricks to take big models. And, you know, take a very particular distribution of user input and train a model that is almost as good as the big model, but much, mu…”

“It's just impossible. It's kind of like piracy, right? I mean, you can sort of all kinds of terms of use, but it's impossible to control distillation.”

“The distillation doesn't do quite as well as the training. And, but neither of them do as well as the diffusion teacher. Including the one that was trained from scratch.”

“For example, all this proprietary stuff sucks at clocks, so nothing that's a distillation will be good at clocks. Nothing can point if you just distill from this. If they can't point, your VLM won't point, so we show how to get good data.”

“There's this whole field of distillation where, you know, Sam can go build the biggest, smartest model in the world, and then you can walk up as a startup and kind of do a distilled version of it and get a model very, very smart at a lot less cost.”