scaling laws

“I don't think we know these things nearly as well as they're known in text. We have some notions of some of the scaling laws here, but I think yeah, we're just so, so far behind.”

“So by doing that, we can see the scaling laws or not just, you know, the scaling laws from like the, you know, chinchilla paper, the scaling laws for all parameters. We can see how does the number of layers change with this? How does the You know, the learning…”

“Chinchilla paper is compute optimal training, but what is not stated in there is it's pre-trained compute optimal training. And once you start caring about inference, compute optimal training, you have a different scaling law and in a way that we did not know …”

“And even if you go and you look at the scaling laws work from Kaplan and Chinchilla and all these other things, they all assume IID data which is insane. We know that all data are not created equal, that garbage in garbage out is like the oldest adage in compu…”

“And what that paper showed was that if you use your data correctly, you can actually bend the scaling laws themselves.”

“This is probably also a unique time in that for the first time you can actually trace dollars to outcomes, right? Provided that scaling laws are holding and capabilities are actually moving forward. Because if you can put translate dollars into capabilities ca…”

“Well, I'll take that. I believe in scaling laws, so”

“And so I think it's just more and more of the same, right? Like more careful research engineering, more careful data engineering, more careful scaling, and it always unlocks that next ability to scale further. So I mean, it's held for You know, almost 10 order…”

“And what we missed throughout that entire journey that we were on the right track, but we should have just kept scaling up. And today to all of us, the scaling laws and scaling up seems like the most obvious thing, but having spent four or five years of my lif…”

“What we are seeing is at Simile, so we do post-train our own model. The thing that we're actually seeing is the early glimpse of scaling law in simulations. The more data about humans and more compute you ingest, you actually start to get predictive and predic…”