Chollet: Gradient Descent Fails at Reasoning by Defaulting to Pattern Matching
François Chollet · François Chollet: Why Scaling Alone Isn’t Enough for AGI · Y Combinator · Mar 27, 2026 · at 14:11
François Chollet, creator of Keras and the ARC-AGI benchmark, discusses the fundamental optimization limits of deep learning when applied to reasoning and theorem proving.
0:00 / 0:31exact quote · 31.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“You could not really get Gradient descent to encode sort of like reasoning style algorithms. It was not because the models could not represent these algorithms. It was because gradient descent could not find them, right? So the problem was that it wasn't about deep learning not being trained complete or anything like that. Like that was not the problem. The problem was gradient descent, right? Gradient descent would not find generalizable programs. It would instead end up doing overfit pattern matching.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from François Chollet
Chollet: AGI Codebase Will Be Under 10,000 Lines on 1980s Compute
“I do believe that, you know, when you create a GI retrospectively, it will turn out that it's a code base that's less than 10,000 lines of code. And that if you had known about it back in the 19 eighties, you could have done a GI back then using the computer r…”
Chollet: Symbolic Models Will Eventually Replicate and Outperform Deep Learning
“And so everything you're doing with machine learning today, with parametric curves, we should be able to do it. With symbolic models in the future in a way that will be much, much closer to optimality. Much closer to optimality in the sense that you're going t…”
Chollet: AI in 50 Years Will Not Use Today's LLM Stack
“I personally don't think that machine learning or AI in 50 years is still going to be built on this stack.”
Chollet: AI field is obsessed with idea that scale alone yields AGI
“Our field became obsessed with the idea that general intelligence would spontaneously emerge by cramming more and more data into bigger and bigger models.”
Chollet: All inventive AI systems rely on discrete search
“All known AI systems today that are capable of some kind of invention, some kind of creativity, they rely on discrete search.”
Chollet: Current LLM Stack Can Fully Automate Any Formally Verifiable Domain
“And I think right now we're in this situation where any problem where the solutions you've proposed can be formally verified, and you can actually trust the reward signal. It's not just some guess made by a model. Any domain like this can be fully automated wi…”