scaling laws

“I think we are on track, even if there were no algorithmic improvements from here, even if we just scaled up what we had so far, ah, I think the scaling laws are going to continue, and I think that's going to lead to amazing improvements.”

“Our view was that, look, this is the beginning of something amazing because there's no limit and you can continue to scale it up. And there's no reason why the patterns we've seen before won't continue to hold. The objective of predicting the next word is so r…”

“I think my basic view is that inference will not get that much more expensive. The simple, you know, the basic logic of the scaling laws is that if you increase compute by a factor of n, you need to increase data by a factor of square root of n. And size the m…”

“So there isn't any evidence that we will not get much better, much more capable models as we continue to scale them across the axis of data and compute.”

“I don't think anyone's seen like these scaling laws, you know, stop. I think as far as anybody has experimented, stuff just gets, keeps, keeps getting smarter.”

“And I think they're both effectively mathematical properties of the universe. I don't think they're you know, just an experimental observed thing.”

“On the material science side, we're seeing scaling laws within physics, within chemistry both with respect to simulations, with respect to experiment, and it's like the same kind of principles at play and ML.”

“The scaling law idea, which is, you know, it's definitely worth agreeing, taking a moment to note that it is not a law of nature. You know, we do not have a principled reason to believe that scaling is some law that will go indefinitely. All we really know is …”