Chollet: Ten random people with majority voting score 100% on ARC-2
François Chollet · François Chollet: How We Get To AGI · Y Combinator · Jul 3, 2025 · at 17:10
François Chollet describes the empirical baseline testing conducted with 400 non-specialist human subjects on the ARC-AGI-2 benchmark suite.
0:00 / 0:16exact quote · 16.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And all tasks in Arc-II were sold by at least two other people that saw it. And each task was seen on average by about seven people. And so what that tells you is that a group of 10 random people with majority voting would score 100% on Arc-II.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from François Chollet
Chollet: AGI Codebase Will Be Under 10,000 Lines on 1980s Compute
“I do believe that, you know, when you create a GI retrospectively, it will turn out that it's a code base that's less than 10,000 lines of code. And that if you had known about it back in the 19 eighties, you could have done a GI back then using the computer r…”
Chollet: Symbolic Models Will Eventually Replicate and Outperform Deep Learning
“And so everything you're doing with machine learning today, with parametric curves, we should be able to do it. With symbolic models in the future in a way that will be much, much closer to optimality. Much closer to optimality in the sense that you're going t…”
Chollet: AI in 50 Years Will Not Use Today's LLM Stack
“I personally don't think that machine learning or AI in 50 years is still going to be built on this stack.”
Chollet: Gradient Descent Fails at Reasoning by Defaulting to Pattern Matching
“You could not really get Gradient descent to encode sort of like reasoning style algorithms. It was not because the models could not represent these algorithms. It was because gradient descent could not find them, right? So the problem was that it wasn't about…”