The Ledger, every show

“And I don't think we're headed to that world largely because of the fact that the models rely so much on large scale test time compute. In order to achieve their greatest intelligence. If you, if it requires so much test time compute to unlock the full capabil…”

“What we're seeing today with the modern models is that 5.5 and other models can think for, if you scaffold them reasonably well, can think for weeks even before having performance plateau on some of these benchmarks.”

“And I wouldn't be surprised if, you know, six months or a year from now, the model is able to do zero shot an entire poker solver, basically my entire PhD thesis in one go.”

“We used an internal model at OpenAI a few weeks ago to disprove the unit Erdos unit distance conjecture.”

“Go ahead and like look at all the published work and synthesize that and then try to come up with something novel and it's not able to do it. And I can give it a lot of time and it's still not able to do it.”

“So at this point, like, you know, the truth is, you know, GPT-IV-O and like O-III, these models are like passing the Turing test.”

“We've said pretty openly that we want to move to a world where there is a single unified model. And in that world, you shouldn't need a router on top of the model. So I think that the router issue Will eventually be solved also.”

“Like, we're gonna scale it, sure, we're gonna scale these things up by a few more orders of magnitude, they're gonna become more capable, but we're not gonna see superintelligence from just that. And like, yes, if we had a quadrillion dollars to train these mo…”

“And I think that if you're able to have them cooperate and compete with billions of AIs over a long period of time and build up a civilization essentially, the things that they would be able to Produce and answer would be far beyond what is possible today with…”

“Like we also found in chess and go, we actually ran this experiment. If you do. Just pure supervised learning on a giant data set of human chess and go games. The bot that you get out from that is not an expert chess or go player. Even if it's like conditioned…”

“I think if you were to look at constrained domains certain negotiation tasks, I think that AIs could probably do better than humans in that today. I mean, I'm trying to think of like specific examples, but things like you know, if you wanted to negotiate over …”

“You know, it doesn't seem crazy to me that you could have a model that can prove the Riemann hypothesis within the next five years. If you can solve the reasoning problem in a truly general way.”

“It turned out that 5.5 is just much more efficient with its thinking. If you run it at max settings, 5.4 is thinking for a lot longer. It takes longer to get back a response than 5.5. And once you control for the amount of thinking time, actually you can see t…”

“Actually the AISI in their evaluations has shown that the models continue to improve at A hundred million tokens. You know, if you run them for a hundred million tokens, they're still improving at beyond that point.”

“We're seeing now with the most recent models that you can actually scaffold, for example, 5.5 into doing a series of experiments that can run for weeks, for months.”

“After we announced the results, A bunch of people found that you could get the answer out of 5.5 as well. If, now, it's not as simple as just asking 5.5, hey, here's the Irish unit distance conjecture. What's the disproof? You had to scaffold it a bit. You had…”

“I think that we're going to continue to see, as I said before, that we're going to see this paradigm continue to progress rapidly. And I think that that's true even today, that we saw that with like going from O-one preview to O-one to O-three, consistent prog…”

“How far does O three get without any harness? How far does it get playing Pokemon? And the answer is like, not very far, you know?”

“One of OpenAI's big success was betting on the scaling paradigm. It is just kind of odd because, you know, they were not the biggest lab, you know, it was, like, difficult for them to scale. Back then, it was much more common to do, like, a lot of small experi…”

“I think that Oh, three is not where the technology will be in six months.”

“And my guess is that if somebody put in the effort, they could probably make a superhuman bot for Magic the Gathering now.”

“You can probably easily 10 X that, you know, I wouldn't be surprised if there's a five hundred million dollar model that's trained in the next year or two.”

“I don't think you can get an AI to output like the next Harry Potter just yet. That might not be that far off. Maybe it's like five years away or something. But I don't think it's happening just yet.”

“Like how many games does it take for an AI, for a human to become a good chess player or a good diplomacy player or a good artist? The answer is orders of magnitude less than it takes for an AI.”

“Like next, next token prediction is going to, is getting you surprisingly far. But I don't think it's gonna get you all the way there to like replacing, you know engineers at big companies.”

“If we were to add this search, this planning algorithm that would come up with a better strategy when it's actually in the hand, how much better could it do? And the answer was it improved the performance by about a 100,000 X. It was the equivalent of scaling …”

“Like, with GPT-III, you couldn't scale test time compute. Like, if you gave it a budget of ten million dollars and said, okay, well, let's see what GPT-III can do, it really can't do that much, more than what you could do with, like, 10 dollars or one dollar.”

“I was really impressed with the model's ability to optimize the algorithms that I had developed in my PhD. It was honestly, it was shocking to see how inefficient I was in retrospect, and they were able to make it like, you know, 1000 x faster.”

“With Tic-Tac-Toe, we see that, like, GPD-Four .5 falls over. You know, it plays decently well. I shouldn't say it falls over. It does reasonably well. You can draw the board. It can make legal moves, but it will make mistakes sometimes, and if you really need …”

“I think it was around, like, November, twenty-twenty-three, or October, twenty-twenty-three, when I think I was convinced that we had, like, very conclusive signs of life, that, like, oh, this was going to be, this is the paradigm, and it's going to be a big d…”

“The way the Poker AI's work today, they're just kind of like sticking to their precomputed GTO strategy. And they're not adapting to the other players at the table.”

“The idea of AGI was really science fiction. There were some people that were, you know serious about it, but very few, the majority opinion was that AI was, if anything, it was kind of a dead field.”

“But surprisingly, we managed to go, like, the full 40 games without being detected as a bot.”

“If you take out the planning that's being done in AlphaGo and just use the raw Policy network, the raw neural network, it's actually substantially below top human performance.”

“And that planning algorithm that's used in AlphaGo, Monte Carlo Tree Search, is very domain specific. I think people don't appreciate just how domain specific it is because it works in chess, it works in Go, and these have been like the classic domains that pe…”

“What's really interesting about diplomacy, aside from just the natural language component, is that it really is the first major game AI breakthrough in a game that involves cooperation.”

“Eventually I'd imagine that these systems become so strong that, like, it kind of goes the way of chess, where, like, the human's just kind of, like, adding a marginal difference at the end.”

“We did another competition that bought one and that bought Cost under a 150 dollars to train if you were to run it on like a cloud computing service.”

“And I should also say like the way professional poker players train now, They all use bots to assist them. It's a lot like chess where you play the game and then you have a bot analyze your play at the afterwards and see like, okay, did you make mistakes? Wher…”

“When we released Cicero, we announced it in, like, late twenty-twenty-two, I still found the game, like, really fascinating, and so I, like, kept up with it, I, like, continued to play, and that led to me winning the championship in the World Championship in t…”