AI Scaling

“What's the bottleneck to building more fabs or to building more chips is more fabs, and people just have not built these fabs yet, right? And that's, I think, the big bottleneck now. And that's gonna persist through the end of the decade. Or until AI, you know…”

“And for all of the concern people have about it's gonna, like, top out, or it's slowing down, or whatever, like, no one's been right about that. I mean, sometimes they, it looked like they were for a couple of months as we digested a new model or came to a new…”

“Actually it turns out you can wring more more juice out of the existing Architectures and data. So there's plenty of room, I think, and we're still seeing that in both the pre-training, the post-training, and the thinking paradigms, and also the way that they …”

“So I think there is probably a little bit more juice out of the scalability element to this piece, so hopefully you'll consider making progress there, but it's getting really, really difficult.”

“So we see the progress as being very fast, and the exponential is continuing, and we don't see any diminishing returns.”

“Without being too specific you know, I think, and we already have maybe some, you know, some evidence to suggest that this is another of those problems that is, is not as difficult as it seems. That will fall to scale Plus a slightly different way of thinking …”

“Why is Mark Zuckerberg building a two gigawatt data center in Louisiana? Why is Amazon building these multi gigawatt data centers? Why is Google, why is Microsoft building multiple gigawatt data centers? Plus buying billions and billions of dollars of fiber to…”

“The thing with AI is if you buy scaling in this picture, that is, you make the models bigger, they get much smarter. Then I don't think it makes sense to hedge your bets in this way. I think you should just double down, give, give one of them a hundred billion…”

“It would just be bizarre to me that, like, you're halfway through the human range of intelligence, and now it stops getting better, so I do sympathize with Sam's statement in the sense of, like, why would it stop here, right? If it was gonna stop, why, it woul…”

“So the most near term limit to scaling is obviously data. This is well known and some research is required to address it. Without going into the details, I'll just say that the data limit can be overcome and progress will continue.”

“We're getting more compute. We can scale up the models more and we can greatly increase the amount of data that, that is programming. So, so, so we have so many ways that we can amplify this. And so Of course it's gonna work. It's just a matter of time.”

“The way I think about it is that we are still pretty far away from the level of scale that is the human brain. And so there's no reason for there to be an asymptote. I think it just keeps working.”