The Wisdom Wall

“Like fundamentally, I think alignment and post training doesn't really make sense as a long-term solution. If you can easily align a model through post training, you can easily misalign a model through post training. If it's easy to put it in, it's easy to take it out. If it's really hard to put it in, it's really hard…”

“basically, you know, when you get to enough scale, inductive biases matter not at all. All that really matters is the learned posterior from the data distribution, and that's really what defines everything.”

“most of what we do in post-training is better were done in pre and mid training and earlier on in training in general.”

“if you filter the data after each point, that's now information injection, and that can break all of this, um, and I think can prevent model collapse.”

“because the model that's doing the rephrasing just needs to know how to rephrase. It doesn't need to know anything about the content itself. It doesn't need to understand it. It means you can use a pretty weak model. To do the rephrasing and have it generalize and generate data that can teach a model that's much better…”

“if there's only one thing that you should take away from this entire interview about what is good for, for data quality, it's diversity.”

“Repeating higher quality tokens is almost always better than seeing net new lower quality tokens. So, like, epoching over higher quality data, almost always better than getting the same amount of new data of an unknown quality, or of average quality, average in this case being like what you just get from an internet…”

“we actually found out that the problem was that the lottery ticket was actually data dependent. And that was where the fundamental problem came. That as soon as you change the data distribution a little bit, like the winning tickets changed in a really big way.”

“Making the data better can be a massive compute multiplier. It can change the performance per dollar by orders of magnitude.”

“the easiest way to think about this is that the value of a data point is not just a function of that data point itself. It's rather a function of how that data point relates to every other data point in the training set, right?”

“Stars are not a good predictor of whether data is useful for models or not. Um, like, I think that's, like, the most popular repos are not necessarily higher quality, at least with respect to do they improve a model's coding capabilities.”

“when you think about the total cost of ownership of these models, it's gonna be very, very heavily weighted towards inference. It's all inference.”

“test time compute as a paradigm really pushes you towards smaller models, right? Because if your cost of, of solving a problem is cost of inference times number of thinking steps, and you have to do a lot of thinking steps. Well, now this is like a really like minimizing the cost of inferences is really important. And…”

“if we apply our curation on top of, um, say DCLM, and then we apply it on top of FineWeb, the gap between FineWeb and DCLM is maintained, um, in the gap between kind of Datology curated DCLM and Datology curated FineWeb. Um, they both get a lot better, but Datology DCLM is still better than Datology FineWeb.”

“In many ways, I think that's kind of the NP-complete problem of AI. Um, if you can do that, you can kind of do anything”

“There are a ton of choices you would make in that process, ranging from how you're going to filter the data, how you're going to sequence the data, what synthetic data you're going to generate, if any, um, how you're going to batch the data, all of those things. And those will have a tremendous impact on the…”

“if you talk to the most talented AI researchers and you ask them, what's the secret to your success, they'll largely tell you that they look at the data.”

“data creation also is a hard problem to solve quote unquote, because it's not one where there's a single silver bullet. There's not just do this one trick and all of a sudden things work. It's rather here are these 50 different things that you can do, each of which provides a pretty modest gain on its own. But then if…”

“The amount of data that I need in order to properly understand dogs is going to be a lot higher than the amount of data I need to understand elephants.”

“Power law scaling is terrible. It means that every time you 10 X your data, you get a diminishing marginal return on performance.”

“when you take these different techniques and you try to make them work together, they don't, generally. You can make them work together, but it's quite hard to do so.”

“there's no golden curation. A curation is only optimal with respect to a given set of downstream use cases or tasks, right?”