scaling laws

“And if you see there are starting to see new kinds of architectures that are going to change how these models scale in the future. And I suspect, ah, that is going to change, ah, how These scaling laws are going to be reshaped as well when it comes to that is …”

“And I think what you're seeing with scaling is we will be able to certainly accomplish our collective definition of super intelligence. With the current, ah, the current path we're on with scaling laws.”

“Our scaling laws still hold. Empirically, there's no reason to believe that there's any kind of diminishing return on pre-training and on post-training.”

“The scaling laws, I think, paint a picture of what is possible, but is not predetermined. Like, to actually get there, there's a lot of, actually, really difficult, both machine learning and engineering work”

“Every time we've kind of run into a like, oh, this isn't quite scaling the way we expect, it's we have a problem. We have a bug that our math wasn't quite right, that, oh, our implementation Isn't quite matching the math, whatever the thing is.”