Scaling Laws

“What we are seeing is at Simile, so we do post-train our own model. The thing that we're actually seeing is the early glimpse of scaling law in simulations. The more data about humans and more compute you ingest, you actually start to get predictive and predic…”

“As you add in more compute and data, what's called loss, aka the loss from next token prediction goes down. And so it's a very smooth linear curve of, like, the models get more intelligent as you scale them up. What's actually also interesting in that paper is…”

“And what we missed throughout that entire journey that we were on the right track, but we should have just kept scaling up. And today to all of us, the scaling laws and scaling up seems like the most obvious thing, but having spent four or five years of my lif…”

“Every time we've kind of run into a like, oh, this isn't quite scaling the way we expect, it's we have a problem. We have a bug that our math wasn't quite right, that, oh, our implementation Isn't quite matching the math, whatever the thing is.”

“And so I think it's just more and more of the same, right? Like more careful research engineering, more careful data engineering, more careful scaling, and it always unlocks that next ability to scale further. So I mean, it's held for You know, almost 10 order…”

“The scaling laws are continuing to hold, meaning bigger model with more data, with more compute, means that the models smoothly keep getting better. It's like the, when the pre-training loss keeps coming down, it's like the model's performance on also all thes…”

“Well, I'll take that. I believe in scaling laws, so”

“From what we see, the scaling laws are not slowing down.”

“But ultimately, yes, Mythos is a significantly larger model. It's proof that the scaling laws still work. Everything about it shows that the trend line continues of models. More compute into model makes model better.”

“You know, betting that scaling laws are gonna stop is, like, a really low probability assumption. I mean, there's just nothing to suggest that's the case. In fact, everything suggests the opposite”

“It has to be the case that if you believe in scaling laws and you believe in AI, that economic growth will be very powerful. I mean, this is the ultimate productivity tool. And what productivity does is allows you to grow while having disinflation, which is li…”

“If you scale up models, you give them more data, more compute. Again, there are a few modifications like RL, but not really very much. It's pretty close to pure scaling. You find that, you know, when you do that, you find, you know, incredible increases in per…”

“And so the scaling laws just tell you that, like, the, you know, if you put in the ingredients to the chemical reaction, the ingredients of data and model size, that what you get out is, is intelligence. Intelligence is the product of the chemical reaction.”

“This is probably also a unique time in that for the first time you can actually trace dollars to outcomes, right? Provided that scaling laws are holding and capabilities are actually moving forward. Because if you can put translate dollars into capabilities ca…”

“It turns out that the scaling laws are proportional to the amount of data you can train on. So you can't just arbitrarily make the model super big if the data set is just the internet. Yeah, maybe you can accentuate that with other kinds of like proprietary in…”

“We'll find out in the next, you know, two, three years if that's really happening, if it's really slowing down. So far it hasn't.”

“I, I'm not sure if I would have bet a lot on, on that actually materializing and being where we are today.”

“Well, I do think Gemini three was very important because it showed us that scaling laws for pre-training are intact.”

“No one on planet Earth knows how or why scaling laws for pre-training work. It, it's actually not a law, it's an empirical observation, and it's an empirical observation that we've measured extremely precisely and has held for a long time, but our understandin…”

“Based on the scaling laws of pre-training, there really should have been no progress in 24 and 25.”

“And so all the progress we've had immense progress since October, 24 through today was based entirely on these two new scaling laws.”

“And we now know they're going to continue for pre-training for at least one more generation, and we're very early in the two new scaling laws, you know, for post-training, mid-training, RLVR, whatever people want to call it, and then test time computed inferen…”

“What scaling clause says is that your loss will log linearly decrease with your compute. We totally see that and clearly Google sees that and all other labs.”

“I think scaling laws of more data, more GPUs and bigger current model architecture is there's still a lot to be done there, but I absolutely think we need to innovate more.”

“I think we don't have any good scaling laws yet. That tell us the trade off, especially I think because it's very hard to measure what is the quality of a data point, right? Like how good is this example compared to this other example without being able to mea…”

“The scaling law idea, which is, you know, it's definitely worth agreeing, taking a moment to note that it is not a law of nature. You know, we do not have a principled reason to believe that scaling is some law that will go indefinitely. All we really know is …”

“The scaling laws, I think, paint a picture of what is possible, but is not predetermined. Like, to actually get there, there's a lot of, actually, really difficult, both machine learning and engineering work”

“On the material science side, we're seeing scaling laws within physics, within chemistry both with respect to simulations, with respect to experiment, and it's like the same kind of principles at play and ML.”

“There's this idea of scaling laws, which is that you can actually quantify, like, as you put in more compute, more, more data, more parameters, you get models in a very, you got a lower loss, a better prediction of the next word in a very predictable way.”

“You can get pretty big wins from RL. You sort of have another set of scaling laws. It's like you put more and more compute into RL, you can get better and better models out of that.”

“And so thinking, post-training, pre-training, we now have three scaling laws, not one.”

“And even if you go and you look at the scaling laws work from Kaplan and Chinchilla and all these other things, they all assume IID data which is insane. We know that all data are not created equal, that garbage in garbage out is like the oldest adage in compu…”

“And what that paper showed was that if you use your data correctly, you can actually bend the scaling laws themselves.”

“Our scaling laws still hold. Empirically, there's no reason to believe that there's any kind of diminishing return on pre-training and on post-training.”

“You can see scaling laws in the reinforcement learning phase of AI training.”

“I think that scaling Really suggests a kind of smooth curve towards what I expect is kind of human level AI or AGI.”

“I think with scaling laws, the holy grail is finding a better slope to the scaling law, because that means that as you put in more compute, you're going to get a bigger and bigger advantage over other AI developers.”

“If scaling laws are failing, it's because we've screwed up AI training in some way. Maybe we got, ah, we got the architecture of the neural network wrong, or there's some bottleneck in training that we don't see, or there's some problem with Precision and the …”

“If you look at the scaling laws, they're continuing to hold true. We did kind of need this transition from like normal pre-training to reinforcement learning, scaling up to continue the scaling laws.”

“And I think what you're seeing with scaling is we will be able to certainly accomplish our collective definition of super intelligence. With the current, ah, the current path we're on with scaling laws.”

“Then this inference time compute seems to have really added in another massive scaling law.”

“We've, we're seeing evidence of scaling laws in biology across proteins, right? That's been shown in the protein language models across DNA, which is what we've shown in our EVO series of models from ARC. We're also seeing inference time scaling laws which in,…”

“They're misunderstood because the assumption is that all of the data is the same quality.”

“And I think they're both effectively mathematical properties of the universe. I don't think they're you know, just an experimental observed thing.”

“Just getting to that point where it's sort of plausibly begins to work is, is a huge, difficult problem. And it's completely separate from using scaling laws. Now, once you get it to work, that's when scaling laws come into play.”

“Chinchilla paper is compute optimal training, but what is not stated in there is it's pre-trained compute optimal training. And once you start caring about inference, compute optimal training, you have a different scaling law and in a way that we did not know …”

“We're starting to enter into a sort of flat part of the curve and we're certainly past the point where if you just interact with a model, You can know how smart it is. Like the vibe checks, they're losing utility.”

“But I think there is no denying that you are, we're on a scaling law right now where orders of magnitude matter. The next model is going to be an order of magnitude bigger and the next one on and on. And so that does make it very capital intensive.”

“I think we're going to see incredible progress in scaling laws in just these new modalities.”

“When people talk about the scaling loss in neural networks, the scaling laws are actually a to a large extent of a property of the transformer. Before the transformer, people were playing with LSTMs and stacking them, etc. You don't actually get like clean sca…”

“The evidence today, I think, is more strongly in favor of scaling laws, and I hope to see more evidence of the, of these other breakthroughs being able to drive progress”

“So by doing that, we can see the scaling laws or not just, you know, the scaling laws from like the, you know, chinchilla paper, the scaling laws for all parameters. We can see how does the number of layers change with this? How does the You know, the learning…”

“It's very unclear whether you can really the scaling law is really enough to get Get you to prove Riemann hypothesis or any of the math conjectures. So you know, and also you have to be superhuman performance in some sense, right? So if you turn on just the co…”

“And if you see there are starting to see new kinds of architectures that are going to change how these models scale in the future. And I suspect, ah, that is going to change, ah, how These scaling laws are going to be reshaped as well when it comes to that is …”

“I don't think we know these things nearly as well as they're known in text. We have some notions of some of the scaling laws here, but I think yeah, we're just so, so far behind.”

“The most technical definition of scaling law does apply and we have seen it apply in this domain. And it's somewhat not surprising because like if you think about like the scaling law in the most technical sense, which is if you scale up data and you scale up …”

“I don't think anyone's seen like these scaling laws, you know, stop. I think as far as anybody has experimented, stuff just gets, keeps, keeps getting smarter.”

“So there isn't any evidence that we will not get much better, much more capable models as we continue to scale them across the axis of data and compute.”

“I think we are on track, even if there were no algorithmic improvements from here, even if we just scaled up what we had so far, ah, I think the scaling laws are going to continue, and I think that's going to lead to amazing improvements.”

“Our view was that, look, this is the beginning of something amazing because there's no limit and you can continue to scale it up. And there's no reason why the patterns we've seen before won't continue to hold. The objective of predicting the next word is so r…”