compute

“It's compute first and it's not cheap. It's great talent second. And then of course it's all the normal operating expenses of a more traditional company.”

“There's a rule of thumb that I like, which is basically, you know, if you want to go, if you want to add a nine, if you want to go from 90 to 99% or 99 to 99.9%, That's maybe an order of magnitude increase in compute.”

“The real cost of compute is the cost of ideas that didn't work. All the things you had to do to get there.”

“Sometimes when you have a lot of resources, you don't actually think as carefully and as critically about what runs are going to be doing and so forth, and you end up being sometimes more wasteful of compute than if you were kind of more compute constrained.”

“I think with scaling laws, the holy grail is finding a better slope to the scaling law, because that means that as you put in more compute, you're going to get a bigger and bigger advantage over other AI developers.”

“Basically you can Scale up the compute in both pre-training and RL and get better and better performance. And I think that's sort of the fundamental thing that is driving AI progress. It's not that AI researchers are really smart or they suddenly got smart. It…”

“The thing we, like, most need is engineers. Almost always, like, throughout, like, the entire history of this field. It's, like, the case that you throw more compute, the thing kind of works. The challenge is, like, actually doing that.”

“I think that, like, the one sort of general intuition I have is, like, compute is the thing that matters. So, like, I think if you throw enough compute at any of these objectives, you're gonna get something that's probably pretty good, and can kind of be fine …”

“Like, nobody has enough spare compute to just make two, you know frontier models at maximum size, right, with different attributes, so that's pretty difficult.”