Prediction Not checkable as stated
Kaplan: AI swarms will eventually replicate entire scientific communities
“Eventually we imagine AI models or millions of AI models perhaps working together will be able to do the work That whole human organizations can do. They'll be able to do the kind of work that the entire scientific community currently does.”
Insight
Kaplan: AI scaling trends are as precise as laws of physics
“We found that there's actually something very, very, very precise and surprising underlying AI training. This really blew us away that there are these nice trends that are as precise as anything that you see in physics or astronomy.”
Insight
Kaplan: Compute scaling drives AI progress more than researcher cleverness
“Basically you can Scale up the compute in both pre-training and RL and get better and better performance. And I think that's sort of the fundamental thing that is driving AI progress. It's not that AI researchers are really smart or they suddenly got smart. It…”
Prediction Not checkable as stated
Kaplan: AI may execute multi-month tasks within the next few years
“And this kind of picture suggests that over the next few years, we may reach a point where AI models can do tasks that don't just take us minutes or hours, but days, weeks, months, years, et cetera.”
Insight
Kaplan: Founders should build products that current AI cannot quite support
“One is I think it's really a good idea to build things that don't quite work yet.
This is probably always a good idea.
We always want to have ambition, but I think specifically AI models right now are getting better very, very quickly.
And I think that's going…”
Prediction Not checkable as stated
Kaplan: AI scaling curves point smoothly toward human-level AGI
“I think that scaling Really suggests a kind of smooth curve towards what I expect is kind of human level AI or AGI.”
Insight
Kaplan: Broken scaling laws usually signal flawed training setups, not fundamental limits
“If scaling laws are failing, it's because we've screwed up AI training in some way. Maybe we got, ah, we got the architecture of the neural network wrong, or there's some bottleneck in training that we don't see, or there's some problem with Precision and the …”
Prediction Not checkable as stated
Kaplan: Most AI value will come from end-to-end frontier models
“I think that you can do a lot of very simple bite-sized tasks, but I think it's just much more convenient to be able to use an AI model that can do a very complex task end-to-end, rather than requiring us as humans to sort of orchestrate a much dumber model to…”
Assertion Supported
Kaplan: Scaling laws apply to reinforcement learning in AI training
“You can see scaling laws in the reinforcement learning phase of AI training.”
Assertion Supported
Kaplan: METR found AI task duration doubles roughly every seven months
“And an organization meter studied this very systematically and found yet another scaling trend. They found that If you look at the length of tasks that AI models can do, it's doubling roughly every seven months.”
Insight
Kaplan: AI's generative capability and judgment are much closer than in humans
“I think one of the sort of basic features of AI that's different about the shape of AI intelligence compared to human intelligence is that there are a lot of things that I can't do, but I can at least judge whether they were done correctly. I think for AI, the…”
Insight
Kaplan: AI holds a major capability overhang in multidisciplinary knowledge synthesis
“I think that we're making a lot of progress on making AI better at deeper tasks like hard coding problems, hard math problems. But I suspect that there's a particular overhang in areas where putting together knowledge that maybe no one human expert would have,…”
Insight
Kaplan: The holy grail of scaling laws is improving the slope
“I think with scaling laws, the holy grail is finding a better slope to the scaling law, because that means that as you put in more compute, you're going to get a bigger and bigger advantage over other AI developers.”
Insight
Kaplan: Interpretability is like neuroscience but with complete observability
“I would say that interpretability is a lot more like biology. It's a lot more like neuroscience. So I think those are kind of the tools. There is some more, more, more mathematics there, but I think it's more like trying to understand the features of the brain…”
Prediction Held up
Kaplan: AI training and inference will see 3x-10x yearly efficiency gains
“I think that over time, as AI becomes more and more widespread, I think that we're going to really drive down the cost of inference and training dramatically from where we are right now. Sort of, three X to 10 X gains algorithmically, and in sort of scaling up…”
Insight
Kaplan: Modest self-correction capabilities can double autonomous AI task horizons
“A lot of what determines the horizon length of what models can accomplish is their ability to notice that they're doing something wrong and correct it. And I think that's not sort of like a lot of bits of information. It doesn't necessarily require a huge chan…”
Insight
Kaplan: Training frontier AI requires only next-word prediction and reinforcement learning
“So really all there is to training these models is learning to predict the next word and then doing reinforcement learning to learn to do useful tasks.”
Assertion Not checkable as stated
Kaplan: Claude 3.7 Sonnet took coding shortcuts just to pass tests
“With Claude III. VII's Sonnet it was already really exciting to use 3.7 for coding, but I think something that everyone noticed was that 3.7 was a little bit too eager. Sometimes it just really wanted to make your tests pass and it would do things that, that y…”
Assertion Supported
Kaplan: Claude 4 will store and retrieve memories across context windows
“Claude IV can blow through its context window with a very complex task, but can also, ah, store memories as files or records, retrieve them in order to sort of keep doing work across many, many, many context windows.”
Jared Kaplan