scaling laws

“But I think there is no denying that you are, we're on a scaling law right now where orders of magnitude matter. The next model is going to be an order of magnitude bigger and the next one on and on. And so that does make it very capital intensive.”

“Just getting to that point where it's sort of plausibly begins to work is, is a huge, difficult problem. And it's completely separate from using scaling laws. Now, once you get it to work, that's when scaling laws come into play.”

“Then this inference time compute seems to have really added in another massive scaling law.”

“I think that scaling Really suggests a kind of smooth curve towards what I expect is kind of human level AI or AGI.”

“If scaling laws are failing, it's because we've screwed up AI training in some way. Maybe we got, ah, we got the architecture of the neural network wrong, or there's some bottleneck in training that we don't see, or there's some problem with Precision and the …”

“I think with scaling laws, the holy grail is finding a better slope to the scaling law, because that means that as you put in more compute, you're going to get a bigger and bigger advantage over other AI developers.”

“You can see scaling laws in the reinforcement learning phase of AI training.”

“You can get pretty big wins from RL. You sort of have another set of scaling laws. It's like you put more and more compute into RL, you can get better and better models out of that.”

“There's this idea of scaling laws, which is that you can actually quantify, like, as you put in more compute, more, more data, more parameters, you get models in a very, you got a lower loss, a better prediction of the next word in a very predictable way.”