Continuous knowledge consolidation remains unsolved in LLM research and development
Andrej Karpathy · Andrej Karpathy: Software Is Changing (Again) · Y Combinator · Jun 19, 2025 · at 17:02
Andrej Karpathy explains to AI founders that LLM context windows only act as temporary working memory rather than real-time continuous learning.
0:00 / 0:17exact quote · 18.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“LLMs don't natively do this, and this is not something that has really been solved in the R&D of LLMs, I think. And so context windows are really kind of like working memory, and you have to sort of program the working memory quite directly because they don't just kind of like get smarter by, ah, by default, and I think a lot of people get tripped up by the analogies, ah, in this way.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Andrej Karpathy
LLMs invert historical tech diffusion by reaching consumers before enterprise
“LLMs, like, flip, they flip the direction of technology diffusion, ah, that is usually, ah, present in technology. So, for example, with electricity, cryptography, computing, flight, internet, GPS, lots of new transformative technologies that have not been aro…”
Waymo autonomous vehicles still rely heavily on human teleoperation
“Like, you may see Waymo's going around, and they look driverless, but, you know, there's still a lot of teleoperation and a lot of human in the loop of a lot of this driving.”
2025 is not the year of AI agents, it is the decade
“When I see things like, oh, twenty-twenty-five is the year of agents, I get very concerned, and I kind of feel like, you know, this is the decade of agents, and this is going to be quite some time. We need humans in the loop.”
Developers should build Iron Man suits, not fully autonomous robots
“Working with valuable LLMs and so on, I would say, you know, it's less Iron Man robots and more Iron Man suits that you want to build. It's less like building flashy demos of autonomous agents and more building partial autonomy products”
LLMs establish Software 3.0 where natural language prompts are programs
“And I think what's changed, and I think is a quite fundamental change, is that neural networks became programmable. With large language models. And so I see this as quite new, unique. It's a new kind of a computer. And so in my mind, it's worth giving it a new…”
LLMs function as CPUs and operating systems orchestrating context memory
“LLMs are kind of like a new operating system, right? So, the LLM is a new kind of a computer. It's setting, it's kind of like the CPU equivalent. The context windows are kind of like the memory, and then the LLM is orchestrating memory and compute for problem …”