The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 43 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 4 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Chollet: AGI Codebase Will Be Under 10,000 Lines on 1980s Compute
“I do believe that, you know, when you create a GI retrospectively, it will turn out that it's a code base that's less than 10,000 lines of code. And that if you had known about it back in the 19 eighties, you could have done a GI back then using the computer r…”
Chollet: Symbolic Models Will Eventually Replicate and Outperform Deep Learning
“And so everything you're doing with machine learning today, with parametric curves, we should be able to do it. With symbolic models in the future in a way that will be much, much closer to optimality. Much closer to optimality in the sense that you're going t…”
Chollet: AI in 50 Years Will Not Use Today's LLM Stack
“I personally don't think that machine learning or AI in 50 years is still going to be built on this stack.”
Chollet: Gradient Descent Fails at Reasoning by Defaulting to Pattern Matching
“You could not really get Gradient descent to encode sort of like reasoning style algorithms. It was not because the models could not represent these algorithms. It was because gradient descent could not find them, right? So the problem was that it wasn't about…”
Chollet: AI field is obsessed with idea that scale alone yields AGI
“Our field became obsessed with the idea that general intelligence would spontaneously emerge by cramming more and more data into bigger and bigger models.”
Chollet: All inventive AI systems rely on discrete search
“All known AI systems today that are capable of some kind of invention, some kind of creativity, they rely on discrete search.”
Chollet: Current LLM Stack Can Fully Automate Any Formally Verifiable Domain
“And I think right now we're in this situation where any problem where the solutions you've proposed can be formally verified, and you can actually trust the reward signal. It's not just some guess made by a model. Any domain like this can be fully automated wi…”
Chollet: LLM Progress in Non-Verifiable Domains Will Slow or Stall
“Progress of reasoning models and base LLMs on this type of domain is, is, you know, it's going to be very slow because the stack we're using, like the LLM stack is very, very reliant on its trained data. It's basically just operationalizing the trained data. A…”
Chollet: General Intelligence Is Human-Level Skill Acquisition Efficiency
“General intelligence is human level. Skill acquisition efficiency on the same scope of tasks that humans could potentially learn to do.”
Chollet: Future AI Won't Just Be Layered Harnesses on Base LLMs
“Future AI in a few decades it's not going to be this harness on top of a reasoning model on top of a base LLM.”
Chollet: Human-Engineered Agent Harnesses Prove AI Is Far From AGI
“I mean, to me, the fact that you need humans to engineer these harnesses is also a sign that we're short of AGI today, because if we had AGI, you know, AGI would just make its own harness. It would not need to be told how to solve a problem.”
Chollet: RL Benchmarks Like Dota and Atari Test Memorization, Not Intelligence
“If you look at Atari games, for instance, or even Dota, you're training on, on the same environment as what you use for testing. So effectively, you're just trying to memorize the best strategies. You're trying to at training time, explore the full space of po…”
Chollet: AI built from first principles will be more efficient than human brains
“It's an implementation of fundamental principles, the fundamental principles of intelligence, which, you know, I think we can identify these principles and re-implement intelligence from scratch, from first principles, in a way it will be much more efficient t…”
Chollet: Replicating Biologically Plausible Brain Mechanics for AI Is Counter-Productive
“I think it would be counter-productive to just try to, you know, observe it and re-implement it, like and make it biologically plausible.”
Chollet: Human generalization stems from symbolic, causal program synthesis
“I do believe the human mind does at the highest level something that looks a lot like programs in this, like we're currently building Causal models of our surroundings. Like we are describing our surroundings in our mind as, you know, a set of objects and agen…”
Chollet: Deep learning guidance is necessary to break combinatorial program search
“You have to break the combinatorial wall, and the way to do it is to add deep learning guidance. It's actually very similar to the principles that analyze something like AlphaGo or AlphaZero.”
Chollet: Artificial General Intelligence Will Likely Arrive Around ARC-6 or ARC-7
“My timeline to AGI, you know, if you just try to extrapolate from the current rate of progress and the amount of investment that's going into, not just the LLM stack, but also like side ideas, side bets that might work out, like, you know, India, for instance,…”
Chollet: Scaling genetic algorithms could automate scientific discovery
“If you try to scale up genetic algorithms, I mean, I'm sure you can do incredible things with that. You could, in fact, probably do new science. Because that's based on search, and search is the best fit for automating the scientific method.”
Chollet: AI Architectures Requiring Human Engineers to Scale Will Fail
“If you're working on something, but the only way to increase the capabilities of the system Is to have human engineers and researchers spend time on it. It will not work because even if the idea is very clever and very elegant and works really well, capabiliti…”
Chollet: Scaling LLMs 50,000x Only Lifted ARC Accuracy to 10%
“And from at the time, back in 2019 to now, with a model like GPT 4.5, for instance, there's been a roughly 50,000 X scale up of basal alarms. And we went from zero percent accuracy on that benchmark to roughly 10%, which is not a lot.”
Chollet: Displaying skill across tasks does not demonstrate intelligence
“Intelligence is a process, and skill is the output of that process. The skill itself is not intelligence, and displaying skill at any number of tasks does not show intelligence.”
Chollet: Human exam benchmarks cannot measure progress toward AGI
“And that's the reason why using exam-like benchmarks with AI models is a bad idea. They're not going to tell you how close we are to AI. Because human exams weren't designed to measure intelligence. They were designed to measure task-specific skill and knowled…”
Chollet: Base LLMs score 0% and static reasoning scores 1-2% on ARC-2
“Well, if you take Bazel Alums, model Slack, GPT-IV-IV-V, LAMA-IV, it's simple, they get zero percent. There is simply no way to do these tasks simply via memorization. Next, if you look at static reasoning systems, so systems that use a single chain of tasks t…”
Chollet: Gradient descent requires 3 to 4 orders of magnitude more data than humans
“Gradient descent requires vast amounts of data to distill simple abstractions. Many orders of magnitude more data than what humans need. Roughly three to four orders of magnitude more.”
Chollet: SOTA test-time adaptation takes thousands in compute to solve ARC-1
“Even the latest set of the art CTA techniques they still need thousands of dollars of compute to solve arc one at human level. And that doesn't even scale to arc two.”
Chollet: AI cannot reach high capability without combining Type 1 and Type 2 systems
“I really don't think that you're going to go very far if you go all in on just one of them, like all in on type one or all in on type two. I think that if you want to really unlock their potential, you have to combine them together, and that's what human intel…”
Chollet: AI will evolve into meta-learners synthesizing software on the fly
“AI is going to move towards systems that are more like programmers that approach a new task by writing software for it. And when faced with a new task, your programmer like MetaLearner will synthesize on the fly a program or model that is adapted to the task.”
Chollet: Parametric Learning Cannot Find Minimum Description Length Models
“You know, the minimum description length principle that the model of the data that is most likely to generalize Is the shortest. And I think you cannot find a model like this. If you're doing parametric learning, you need to try symbolic learning.”
Chollet: NDEA Has Only a 10% to 15% Chance of Success
“Like we have maybe a 10 or 15% chance of success. But that is enough that it's worth trying, right?”
Chollet: Mathematics AI Revolution Is Coming in the Next Few Years
“I think mathematics is also, it's also primed to see a revolution in the next few years for the same reasons, again, because The domain just gives you verifiable rewards.”
Chollet: Base LLMs Score Under 10% on ARC-AGI-1
“So basal alarms were scoring extremely low on V-one, like sub-ten percent, basically. And, I mean, it was true of the original, like, GPT-III actually scoring zero, but that's even true of the latest basal alarms today, you know, as of March.”
Chollet: Competency Balances a Trade-Off Between Intelligence and Knowledge
“When it comes to Competency. There's always a trade-off between intelligence and knowledge. If you have more knowledge, if you have better training, you need less intelligence to be competent.”
Chollet: ARC-3 private test set differs substantially from public set
“We've deliberately tried to create a private set of environments that is significantly different from the public set. Like you can look at the public set. It's not actually giving you that much information about what's in the private set. In the private set, y…”
Chollet: Science Is Symbolic Compression, Not Merely Curve Fitting
“Science is not about curve fitting. Science is about finding the equation, finding the most compressive symbolic model of your pile of observation.”
Chollet: Equal investment in genetic algorithms would have yielded exciting results
“If you had thrown the same amount of investment into almost anything else, you would also have seen extremely exciting results, like genetic algorithms, for instance.”
Chollet: AI Empowers Workers With Deep Expertise Rather Than Replacing Them
“The more, you know, the more expertise you have, but things like programming, for instance, the better you're able to Use and leverage these tools for your own benefit. And with the right kind of expertise all this AI progress is actually empowerment. Like, it…”
Chollet: Fine-Tuned OpenAI o3 Reached Human-Level Performance on ARC
“So in particular, in December last year, OpenAI previewed its, ah, all three model, and they used a version of it that was, ah, fine-tuned specifically on Arc, and that showed human-level performance on that benchmark versus time.”
Chollet: Ten random people with majority voting score 100% on ARC-2
“And all tasks in Arc-II were sold by at least two other people that saw it. And each task was seen on average by about seven people. And so what that tells you is that a group of 10 random people with majority voting would score 100% on Arc-II.”
Chollet: Every High-Performing ARC AI Method Uses Test-Time Adaptation
“And today, every single AI approach that performs well on Arc is using one of these techniques.”
Chollet: All ARC-3 environments are solvable by untrained humans
“All of these test environments in Arc three Are solvable by humans with no prior training because we actually tested them on, on regular people.”
Chollet: ARC Prize Built a Video Game Studio Generating 250+ Games
“We set up an entire video game studio, right, to create them. So we got over 250 games.”
Chollet: ARC-4 will focus on continual learning and compounding levels
“So there will be ARC four, which will be in the spirit of ARC three, but more focused on continual learning and curriculum learning at longer timescales. So you're gonna have fewer games but they're gonna have way more levels. And the levels are going to be co…”