Everything Gary Marcus said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Marcus: Pure transformer models lack mechanisms for truth and inherently hallucinate
“If you look at a completely pure case of a transformer model trained on a bunch of data, It doesn't have any mechanisms for truth. Now, except the sort of accidental contingency, and there are inherent reasons why these systems hallucinate, and maybe I can, in…”
Marcus: LLMs hallucinate because they cannot distinguish individuals from categories
“The way I think about it is that these things don't understand the difference between individuals and kinds. So I actually wrote about this twenty-some years ago in my book, The Algebraic Mind, and I gave an example there, which is I said, suppose it was a dif…”
Marcus: ChatGPT succeeded over Galactica due to RL guardrails
“Part of the reason why ChatGPT succeeded where Galactica didn't is Galactica didn't really have those guardrails at all, and so it was just, you know, very easy to get it to say terrible things, and it's harder to get ChatGPT to say terrible things because tha…”
Marcus: We Lack the Science or Engineering to Know Proper AI Controls
“We don't even know what the right controls should be. They didn't do a good job and we don't yet have a good science or engineering practice on what it should be.”
Marcus: AI chatbots create illusions too compelling for average people to discern
“They're too compelling in that illusion for the average person with no training in how they work to understand it.”
Marcus: Current AI is a dress rehearsal showing we are not ready for AGI
“I, I've been thinking about this whole thing as a dress rehearsal. And before we didn't know how to make a dress rehearsal. This is a dress rehearsal for AGI. And you know, the lesson of the dress rehearsal is like, we are not ready for prime time. Let us not …”
Marcus: Machines will convincingly conduct podcast interviews within 30 years
“And some point I'm going to say, 30 years from now. We'll be able to make machines that do podcast interviews and I won't really know. I don't really know if you're a person fake or, you know, machine faking me out or whatever at some point.”
Marcus: No established methodology exists for debugging chatbots or autonomous vehicles
“None of these chatbot systems are well-debugged. Nobody knows how to debug them, in fact. And so both the problem with GPT-III and with driverless cars is we don't actually have a methodology even for debugging it.”
Marcus: Teslas crashed into multiple emergency vehicles due to vision flaws
“It turns out that Teslas have a problem with stopped vehicles, mostly emergency vehicles, on the side of the road. So they have run into, in the last year, three fire trucks, a tow truck a police car, and so forth.”
Marcus: Deep learning requires vast data, quick tasks, and stable domains
“If a typical person can do a mental task with less than one second of thought, and we can gather an enormous amount of data, of directly relevant data, we have a fighting chance. So long as the test data aren't too terribly different from the training data, an…”
Marcus: Deep learning models lack understanding of spatial relationships and silhouettes
“Deep learning doesn't even really fully understand the relations between parts and wholes. It certainly doesn't understand what a silhouette is, and it gets it wrong.”
Marcus: Deep learning systems cannot perform compositionality or combine separate ideas
“Deep learning can't do what we call compositionality. It can't put ideas together.”
Marcus: Deep learning fails at reading comprehension and misses the point
“I think that deep learning doesn't just fail on reading. It completely, entirely misses the point.”
Marcus: General AI requires hybrid models because cognition has distinct components
“There are a lot of different components to cognition, and so we have to have a hybrid model. There's just no way that one system is going to do all of these different things”
Marcus predicts AI developers will increasingly adopt hybrid models in secret
“I suspect that what will actually happen is that people will start to incorporate hybrid models and not call them that.”
Marcus: Current AI systems are fundamentally a collection of idiot savants
“AI is still basically a collection of idiot savants.”
Marcus: AI is stuck due to over-reliance on statistics and big data
“And I have a three-part diagnosis, which is that it's fallen in love with statistics, it's fallen in love with big data, and it's forgotten its roots.”
Marcus: Reinforcement learning fails in physical robotics due to real-world damage costs
“But it's another thing if you import those same techniques into robots. If you have your robots doing reinforcement learning, which is basically trial and error learning, they start knocking over the furniture a 100,000 times, you're probably gonna send it bac…”
Marcus: AI progress will be gradual, not a sudden explosion
“I don't think there's going to be like one special day with this big explosion. They're going to get better and better. It's going to take a while.”
Marcus: OpenAI's o3 hallucinates more than preceding models
“I'll give you just one more example is O three apparently hallucinates more than the models that came before it.”
Marcus: Microsoft study suggests chatbot usage impairs human critical thinking
“Well, Microsoft did a study, in fact, suggesting that critical thinking was getting worse as a function of them.”
Marcus: AlphaFold is a neurosymbolic model and AI's greatest achievement
“And also alpha fold is actually a neurosymbolic model, and it's probably the best thing that AI ever did.”
Marcus: We Have No Way to Truly Debug AI Systems
“I think that's actually the deepest problem here is we have no way to debug these systems, really. We have the band-aids of, like, reinforcement learning and things like that that are so indirect.”
Marcus: LLMs will connect to all major software within two years
“You have companies like Adept that are spending their whole time trying to quote, add all the world, connect all of the world's software to large language models. So if not now, soon enough. I think that that's right. Like, you know, two years if it's not toda…”