The Exchanges

Q So the keywords for that data curation is a service, data efficiency, all those terms. In the pre-chat before we started recording, you mentioned that there's a cool story around how you got into data in the first place, right? You were at GDM. You were at Meta as a research scientist. Describe how like that became an interest.

A My PhD is actually in neuroscience. Uh, so I come much more from an empirical science sort of background. I actually spent time trying to teach mice how to count and then analyze the activity of thousands of neurons in the brain while mice did count and try to understand how did that actually happen? How, what were the neural dynamics that enabled that? Um, and that's actually initially how I got into machine learning was as a means to analyze my, my neural datasets. I also started my PhD, So Alex came right after that, Tari DQN right after that. Lots of evidence that AI was going to be very, very exciting, which, which led to me transitioning. But as a result, because I had this kind of somewhat different background of being trained as an empirical scientist rather than as a computer scientist, my real first mission when I, when I joined AI was to try to build more of a science of deep learning. Something that I think, you know, is still true today in many cases is that deep learning is an empirical science, but most people that have computer science backgrounds were trained more In the context of a branch of theory, right? Everything was very provable. That was the initial pushback to deep learning, actually, was that you couldn't prove anything in it. But deep learning is, at its core, an empirical science, right? We have to run large experiments. We understand the rules for…

AI assessment note: “that's actually initially how I got into machine learning was as a means to analyze”

Q I'll push you a bit on this. Yeah. Um, you know, I think a popular view is post training is elicitation. Of capabilities that you already trained in pre-training. So what dependencies can you have that feedback into the, into pre-training?

A So, so I'm inclined to, to agree with that view. And, and I think that that view would lead very strongly to the fact that you should be trying to optimize your pre-training data to make post-training processes more effective. So you should try to figure out how do I optimize my pre-training data so that the slope of the test time compute curve, or so that the slope of the RL curve is as steep as you possibly can be. Um, or alternatively, how do I optimize my pre-training data so that the slope of the jailbreaking curve Is as shallow as possible, right? Like fundamentally, I think alignment and post training doesn't really make sense as a long-term solution. If you can easily align a model through post training, you can easily misalign a model through post training. If it's easy to put it in, it's easy to take it out. If it's really hard to put it in, it's really hard to take it out. That's just like a truism of models, right? So if you do alignment during pre-training, you'll actually end up with models that are, I think, largely impossible to misalign without putting a massive amount of data into them. Um, I think there are a lot of benefits to that. Um, and I think we've also seen evidence for this, like looking at the difference between Lama and Quen with respect to their ability to be post-trained, right? It's much easier to RL Quen than it is to do Lama. Likely that has t…

AI assessment note: “optimize your pre-training data to make post-training processes more effective”

Q But so what's the right number between one and 10,000?

A The unsatisfying answer is it depends, but it's also the right answer. Um, so it depends on the, on, on how complex the concept is. So redundancy is really useful, right? And, like, removing all redundancy is a bad thing. If I remove all redundancy, then I'd only be able to understand, say, a golden retriever in the one situation that I've ever seen it in before. I wouldn't be able to generalize, and that would be bad, right? So some redundancy is good, but I think we all have the intuitive understanding that infinite redundancy is not good, it's bad. So where is this line for different concepts? Well, one example I like to give for this is elephants versus dogs. So elephants are pretty stereotyped. There are two kinds of elephants in the world. There are Asian elephants and African elephants. Um, they're all gray. They all have floppy ears. They all have a trunk and some tusks. They all have, ah, you know, wrinkly skin. African elephants are bigger than Asian elephants, but largely they're all pretty similar. There's not too much variability. So I don't need that much data or that much redundancy to understand the concept of elephants, you know, fully and completely. But dogs on the other hand are totally different, right? Dogs Are super variable. There are hundreds of breeds, not to mention all the mixes of different dog breeds, their different shapes, sizes, textures, colors…

AI assessment note: “The unsatisfying answer is it depends, but it's also the right answer.”

Q Are you measuring the quality of these open data sets over time? Are the most recent open data sets better than the older ones at a good rate or like just marginal?

A They do get better, but I think they're not relative to the headroom and potential, I would say, right? Like, like, Nematron is actually pretty similar in quality to DCLM. It's, it came out about six months later. It has more unique tokens. Um, they made a really big deal about it having more unique tokens, but on average, the quality is, is pretty straightforward. So, you know, when we think about what we are able to accomplish at Taktology, we usually think about along these three axes I mentioned, train faster, train better, train smaller. So typically, basically that's like first question, train faster. Given a certain baseline dataset, how much faster can we achieve the same performance? So, you know, and how many fewer tokens? Um, so we're able to now get to the same performance as DCLM about 12 x faster. So, you know, in fewer than 10% of the tokens we can match What you get from training to convergence.

AI assessment note: “They do get better, but I think they're not relative to the headroom and potential”

Q Am I right to recreate synthetic data with rephrasing or is there, are there other parts to synthetic data in your mind?

A Yes. I think there are different parts of synthetic data. There are two parts, but let me first actually just comment on the, on the filtering versus things. So I, I used to actually use the word data filtering or data pruning. Um, and I actually, that paper I mentioned that was at NeurIPS, that one actually has data pruning, uh, in the title. And that's how you beat scaling laws through data pruning. When I started Datology, I really changed the language to be data curation. Um, over data pruning or data filtering, and that's because curation is a lot more than just filtering. Filtering and saying, hey, this is a bad data point, we want to get rid of it, is absolutely an important part of what we do, but it's also about rebalancing data sets, upweighting and upsampling certain data distributionally and downsampling others. That might not mean filtering, it might just be changing the, the weighting with, with which you take it. The order in which you present data can be really impactful. Curricula, and we now have seen this with discrete curricula, you know, for multi-phase training and things like that, that's not filtering. You know, the way you batch the data can be an important factor. Synthetic data can be an important factor. The way you mix sources. All of these sorts of things beyond just filtering. So filtering is a very important part of what we do, and it will always…

AI assessment note: “there are two approaches to synthetic data, and we have focused more on one”

Q the next generation of AI, not on this current generation. So my role is the same. But then maybe people might say, you know, then why didn't you do the current generation 10 years ago? What do you make of the whole, of the whole change, and whether or not you think this is an interesting direction for Meta, especially given the large platform and user base that they have?

A Well, first, with respect to Jan, specifically, I mean, Jan's an incredibly talented scientist, of course, but I think that, you know, his preference has always been to do science rather than to run an organization, so I think he ran fair, um, like, organizationally for a year or two right at the very beginning, but pretty quickly, uh, he handed that off to other people, and, like, when I was there, it was Joelle Pinot and, and Antoine Bordes, and then Joelle for most of it, that, that really were running first, and she was an incredible leader. I, I, I really respect her deeply, and, and, Couldn't have asked for a better kind of advocate for science within FAIR.

AI assessment note: “Jan's an incredibly talented scientist, of course, but I think that, you know, his preference”