The Ledger, every show

“And one of my like more controversial viewpoints, I think, is that I think the transformer is a great advance to be sure, but I think it's one of a very large Set of equivalently good architectures that we could have found. And there are many, many ways we cou…”

“And I think that's what we've seen to some extent with the failure of the mega models, right? With 4.5 and Lama four and others. I think that there is a challenge of just continuing to do that naively and you have to figure out how to break it.”

“Like fundamentally, I think alignment and post training doesn't really make sense as a long-term solution. If you can easily align a model through post training, you can easily misalign a model through post training. If it's easy to put it in, it's easy to tak…”

“Turns out in the small data regime, and when I say small data here, I mean, say less than 500,000 data points. And this was in the context of image self-supervised learning. So in that small data regime, this is super helpful. And where this paper's actually b…”

“Basically, you know, when you get to enough scale, inductive biases matter not at all. All that really matters is the learned posterior from the data distribution, and that's really what defines everything.”

“Something I've said before and I'll say again is, is that data is the most under-invested in area of research relative to its impact, and I don't think it's even close.”

“And even if you go and you look at the scaling laws work from Kaplan and Chinchilla and all these other things, they all assume IID data which is insane. We know that all data are not created equal, that garbage in garbage out is like the oldest adage in compu…”

“But I do not think there's any way we could get to where we are today without self-supervised learning and the ability to train on unlabeled data. That was the real advance to my mind that enabled us to get these incredible increases in capabilities.”

“These are nominally the best experts you could ever hire to do this. These are students who have just spent all of their time looking at NLP data for two years. They could not predict what the DCLM classifiers would say above chance.”

“And what that paper showed was that if you use your data correctly, you can actually bend the scaling laws themselves.”

“We're able to now get to the same performance as DCLM about 12 x faster. So, you know, in fewer than 10% of the tokens we can match What you get from training to convergence.”

“You know, we've already been able to get 10 X gains. I think there's at least another hundred X behind this that are still to be done.”

“I think you, what you see in all the frontier labs is that they have data teams. And if you talk to the folks that work on those data teams, what you'll kind of systematically hear is that typically they're under resourced relative to the gains that they're de…”

“Most of what we do in post-training is better were done in pre and mid training and earlier on in training in general.”

“I believe that getting to a frontier model should cost a million dollars or less for most organizations, at least in a specialized domain, right? And when you think about what enterprises need, that's generally what they need. They don't need a model that can …”

“If you filter the data after each point, that's now information injection, and that can break all of this and I think can prevent model collapse.”

“Because the model that's doing the rephrasing just needs to know how to rephrase. It doesn't need to know anything about the content itself. It doesn't need to understand it. It means you can use a pretty weak model. To do the rephrasing and have it generalize…”

“If there's only one thing that you should take away from this entire interview about what is good for data quality, it's diversity.”

“Repeating higher quality tokens is almost always better than seeing net new lower quality tokens. So, like, epoching over higher quality data, almost always better than getting the same amount of new data of an unknown quality, or of average quality, average i…”

“And getting a curriculum right could literally make the difference between, you know, spending 10 times as much on a model training, you know, hundreds of millions of dollars potentially.”

“It's much easier to RL Quen than it is to do Lama. Likely that has to do with the fact that Quen put a lot of synthetic reasoning traces into their training data.”

“The inference costs are going to skyrocket with these models. And if you use a general purpose model, then you constrain to say, hey, this model knows about everything, but now only do this one thing. That model is going to have a ton of parameters that do not…”

“Mosaic was the first one to really recognize that there was a huge opportunity in making this easy. And now this has largely been commoditized by things like SageMaker and Together and lots of different folks that help you on the training side. But on the data…”

“We actually found out that the problem was that the lottery ticket was actually data dependent. And that was where the fundamental problem came. That as soon as you change the data distribution a little bit, like the winning tickets changed in a really big way…”

“Most of the models that the vast majority of people will be using in say three years will be single digit B or smaller.”

“I don't think he was ever you know, or at least not since the beginning in a role where he was defining AI strategy for Meta. I don't think that's the role he wanted at any point. You know, I think he really wanted to be doing that research, and I think, so I …”

“Making the data better can be a massive compute multiplier. It can change the performance per dollar by orders of magnitude.”

“The easiest way to think about this is that the value of a data point is not just a function of that data point itself. It's rather a function of how that data point relates to every other data point in the training set, right?”

“Stars are not a good predictor of whether data is useful for models or not. Like, I think that's, like, the most popular repos are not necessarily higher quality, at least with respect to do they improve a model's coding capabilities.”

“When I was at Meta, certainly legal stuff around data sets was very challenging and becoming increasingly challenging, and there are a number of situations where, you know, the only person that could approve things was Zuck because of the scale of the risk, I …”

“Nematron is actually pretty similar in quality to DCLM. It's, it came out about six months later. It has more unique tokens. They made a really big deal about it having more unique tokens, but on average, the quality is, is pretty straightforward.”

“What we've tried to do, and I think we've done a good job of, and I'm generally happy with the balance we've struck is try to, in the blog posts that we put out, give a lot of intuition as to kind of what we're doing and how it works without necessarily gettin…”

“Something that you don't see happen even at the big labs because they have entirely separate teams, right? There's a free training team. There's a mid training team. There's a post training team. And like the mid training team is a customer of the free trainin…”

“When you think about the total cost of ownership of these models, it's gonna be very, very heavily weighted towards inference. It's all inference.”

“Test time compute as a paradigm really pushes you towards smaller models, right? Because if your cost of solving a problem is cost of inference times number of thinking steps, and you have to do a lot of thinking steps. Well, now this is like a really like min…”

“It was beating Gemma pretty consistently before the one trillion mark, which was pretty cool to see.”

“If we apply our curation on top of say DCLM, and then we apply it on top of FineWeb, the gap between FineWeb and DCLM is maintained in the gap between kind of Datology curated DCLM and Datology curated FineWeb. They both get a lot better, but Datology DCLM is …”

“In many ways, I think that's kind of the NP-complete problem of AI. If you can do that, you can kind of do anything”

“I think the one that's still really up in the air is a metaverse, but I would actually argue that I think that's going to end up paying off in the long run. I think the Ray-Ban glasses pretty darn cool. And a lot of the foundations of what was in reality labs …”

“There are a ton of choices you would make in that process, ranging from how you're going to filter the data, how you're going to sequence the data, what synthetic data you're going to generate, if any how you're going to batch the data, all of those things. An…”

“Help the folks we work with to train models much faster to much better performance and to also help them train much smaller models to the same or better performance, which I actually think is some of the most exciting stuff going forward. But fundamentally, th…”

“In general data work has been far more valued in industry consistently than it had been in the research community.”

“If you talk to the most talented AI researchers and you ask them, what's the secret to your success, they'll largely tell you that they look at the data.”

“Data creation also is a hard problem to solve quote unquote, because it's not one where there's a single silver bullet. There's not just do this one trick and all of a sudden things work. It's rather here are these 50 different things that you can do, each of …”

“The amount of data that I need in order to properly understand dogs is going to be a lot higher than the amount of data I need to understand elephants.”

“Power law scaling is terrible. It means that every time you 10 X your data, you get a diminishing marginal return on performance.”

“We're wasting a ton of capacity in these models on knowledge that is just totally unnecessary for them to have.”

“When you take these different techniques and you try to make them work together, they don't, generally. You can make them work together, but it's quite hard to do so.”