Answered raw tape
D 5 · C 5 · P 4 · Cm 4 4.60
Q in any reasonable quantity. Like, you know, you know, Claude 3.5 Opus, like if it, if it does exist, still not like, you know, the, the thing that we actually use is Sonnet, right? So like, it's almost like a deployment strategy. Like you, you train the large model, the, the teacher model in order to distill. Like, you don't actually expect to use the teacher model for inference anymore.
A Yeah, I think it is really the, the marginal benefit versus cost argument, right? Like, if the marginal benefit and capability is not worth the additional cost to you, then people don't want to use that bigger model. And it is very, very task dependent. Maybe you really want that capability, but sometimes, especially when you're doing things at bulk, maybe it's okay that you're slightly worse off, but it's much cheaper. And that's maybe one of the reasons, at least this graph, that's why you want to be at the Pareto frontier. And I think the other thing is the capability which we have right now, maybe next year will be much cheaper to have that same thing. And that likely is the result of distillation, right? Like that's, that's, it's not just because we are doing or figured out something magical. It is because distillation can help you bring the cost of any existing capability.
AI assessment note: “distillation can help you bring the cost of any existing capability”
Answered raw tape
D 3 · C 3 · P 3 · Cm 2 2.85
Q It's still a lot less efficient than the larger matching distillation, right?
A It depends. I guess I'll, that's a complicated, uh, question. It depends on the thing, but yeah, logits are definitely assuming access to more information. So it, it can be, but then again, you don't have access to logits if you want to distill, let's say, a black box model. So, but then again, no one wants, it's not legal, I guess, to distill the exact model, or at least the terms and conditions don't allow you to do that. But for open models, yeah, you can do better because you have access to logits, so why not use them? At least that's my take. So, um, and I guess this, maybe we can skip, but this is just saying when you're doing best of n, why not keep the top k rather than just the top? Because if you're already generating, let's say, a lot of samples, why, why throw away, let's say, if you have some sort of scoring mechanism of how good something is, and this thing is called MBR in the translation literature, but anyways, this is another way of best of hands, so the key idea is very simple, just like Use some of the, yeah, just don't throw away everything. Just use the top K samples rather than just the top one, and this might be better, and that's true. So that's like a word, a simple thing worth trying if people are doing synthetic data distillation. Maybe, yeah, so there's small gains you can see. MBR is, let's say, using just the top one, and then if you increase from…
AI assessment note: “It depends... for open models, yeah, you can do better because you have access to logits”
