Continual Learning
topic on 7 shows · 22 statements across 17 episodes
the Y Combinator Startup Podcast 
Cheeky Pint 
Latent Space 
the MAD Podcast 
the a16z Podcast 
Big Technology 
20VC
22 statements about Continual Learning, every show
Editing isolated facts in MLP weights fails to update multi-hop downstream reasoning
“So if you know that the best university in the world is Waterloo, then the answer should be Waterloo. But if I wasn't just one-shotting the question and I was to ask it to like, use its knowledge to think and then give me a second answer, or like, should I hir…”
KV cache compaction is the viable path to solve LLM continual learning
“If you're able to sort of make your KV almost infinite, and you're able to compact in such a way that you don't lose any of the knowledge. In that case, you can actually do a continual learning as you can actually solve continual learning. And this, as a, it's…”
Malde: Continual learning will be AI's next major unlock
“And we realized continual learning is kind of the ultimate, like, paradigm to do that. Is like, how do you have humans in the loop? How do you build this intelligence around them that is constantly learning and growing on its own? And I think that's going to b…”
Malde: Standard reinforcement learning is broken for continual learning
“RL, it's still taking all of this kind of Useful information from the real world, like I mentioned, all the corrections and everything, and putting it into just one number. Which is really broken.”
Hassabis: Continual learning, reasoning, and memory are required for AGI
“Continual learning, Long-term reasoning. Some aspects of memory. These are still unsolved. And how to get the systems to be more consistent across the board. I think all of these are going to be required for AGI.”
Hassabis: Lack of continual learning holds AI agents back from full tasks
“I think that's one of the not having continual learning currently is one of the things holding back agents from doing full tasks. You know, I think they're really useful for aspects of tasks right now, and you can patch them together and do some really cool th…”
Hassabis: Current AI systems lack continual learning capabilities after deployment
“These systems don't learn after you finish training them, after you put them out into the world.”
Self-improvement increases AI capability; continual learning keeps its knowledge fresh
“So, self-improvement is about a model getting smarter. Over time and improving its capability, like the model itself doing it. Continual learning is mostly about a model staying current, right?”
Emergent agents learn across user sessions using long-term trajectory memory
“We were able to figure out, okay, all the trajectories that we are generating, we can kind of aggregate over time and like sort of build in a long-term memory for the agent, which is very unique in the sense that your agent learns not just from your own sessio…”
Pineau: AI research community lacks agreed definition of continual learning
“I confess I have a little bit of trouble with continual learning as a concept because I feel the community has never been able to nail, like, how do we articulate the problem in a way that we all agree on it? And so everyone who does work on continual learning…”
Pineau: Continual learning research is disconnected from scaling efforts
“Right now the progress in the research community that's working on continual learning isn't necessarily connecting to the work that's going on on scaling.”
Self-improving AI and continual learning agents will not be feasible in 2026
“If you have an LLM that self improves or like an agent that does something fails and learns, I don't think anything like that is feasible this year.”
Hassabis: Continual learning that updates models over time has not been cracked
“But I think to have it you want to do it more than just having your data in the context window. That's you want to have something a bit deeper than that, which is, as you say, actually changes the model over time. That's what ideally you would have. And that t…”
Izmailov: Current AI models lack continual learning and cross-setting coherent goals
“I am pretty confident we are not there at the moment. I think right now the models are still acting in isolated environments, and we are not seeing a lot of evidence for very coherent goals across different settings”
Nair: Continual learning during deployment does not risk model capacity overload
“If you could learn enough about those million tokens that you're actually in deployment on I don't think you should need, like, I don't think there's a risk of overloading the capacity of your model, right? Because you can train on a trillion tokens, and it's …”
Bourgeau: Recent continual learning progress has mostly occurred via post-training search tools
“First, I think a lot of progress has been made on this front since in the last few years. I think this is mostly around post-training, around search, use search tools and then make search calls, then they would have access to that new information.”
McDermott: Autonomous continual learning creates an unverifiable black box for enterprise support
“When we're changing the machine and we're writing new articles and we're building new macros, it's sort of easy for the humans who own that machine to understand what's going on. Like the head of support technology can be like, yep, yep, that is the correct wo…”
Nadella: Separating cognitive core from knowledge enables continual learning
“If you kind of have the model's cognitive core separated from its knowledge, then you have essentially the continual learning, you know, formula, so to speak, or the algorithm, and then you just unleash it.”
Amodei: Continual learning will be solved by scaling and new methods
“Without being too specific you know, I think, and we already have maybe some, you know, some evidence to suggest that this is another of those problems that is, is not as difficult as it seems. That will fall to scale Plus a slightly different way of thinking …”
Dwarkesh: Lack of continual learning prevents LLMs from replacing human labor
“I think a big bottleneck these models have is their inability to learn on the job, to have continual learning. Their entire memory is extinguished at the end of a session. There's a bunch of reasons why I think this actually makes it really hard to get human-l…”
Patel: Multi-instance continual learning could create superintelligence without recursive coding
“Once continual learning is solved, you might have something that looks like a broadly deployed intelligence explosion, which is to say, That because if these models are broadly deployed to the economy, every copy that's like this copy is learning how to do plu…”