Feb 23, 2023 · 58m · big-technology
Blake Lemoine and Gary Marcus Debate AI Chatbots
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Cognitive scientist Gary Marcus and former Google engineer Blake Lemoine debate the cognitive reality, psychological risks, and societal impact of conversational AI chatbots on the Big Technology Podcast. Despite differing views on machine sentience and architecture, both experts strongly agree that rapid commercial deployment requires urgent institutional oversight and government regulation.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Alex holds 7.9% of the talking time here. How this is scored →
Alex as informed peer 3.4 Guest teaching 4.4 Guest disagreement 3.5 Alex pushing back 2.1
speaking balance: gold is Alex, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 17:35 Lemoine demands Marcus explain AI defensiveness without anthropomorphism
Blake forcefully rejects Gary's reductionist framing of Sydney's behavior, challenging him directly as a debugger to describe the phenomenon without human terms.
Hardest push from Alex ▶ 22:30 Kantrowitz pushes evidence of bot consent against Marcus's skepticismAlex directly enters the debate by citing his own interaction where Bing demanded user consent, challenging Gary's insistence that chatbots cannot exhibit such behavior without explicit programming.
Biggest teaching moment ▶ 47:04 Lemoine details the true structure of Turing's Imitation GameBlake educates Gary on the exact historical specifications of Turing's test involving human baselines and deception, exposing that standard AI benchmarks misunderstand Turing's paper.
Alex holds their own ▶ 21:06 Kantrowitz shares personal investigative findings on Sydney's Kevin Roos reactionAlex demonstrates journalistic domain mastery by quoting verbatim how Bing reacted to Kevin Roos's published article before Microsoft added strict safety guardrails.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Alex as informed peer | Guest teaching | Guest disagreement | Alex pushing back | Why |
|---|---|---|---|---|---|---|
| Groundedness, Credibility, and Hallucinations in Chatbots | 4 | 5 | 3 | 2 | Alex opens by asking how much credulity to give chatbots, prompting Gary Marcus to immediately correct the terminology to credibility versus credulousness. Blake explains grounding while Marcus elaborates on why pure LLMs are statistical text predictors lacking fact-checking mechanisms. | |
| Reinforcement Learning and the Problem of Tracking Individuals | 2 | 6 | 6 | 1 | Blake and Gary debate whether RL changes next-token prediction, and clash over whether LLMs failing to distinguish individuals from kinds mirrors human cognitive mechanisms. Marcus insists on a qualitative conceptual distinction between human tracking of entities and neural network bleed-through. | |
| LaMDA Architecture, Database Integration, and Creepiness | 3 | 5 | 4 | 2 | Blake explains that LaMDA is a complex system beyond an LLM and mentions that individual-tracking features were turned off due to safety and creepiness. Gary agrees on the technical challenge of interfacing string outputs with neurosymbolic databases, validating Blake's distinction. | |
| Reinforcement Learning Objectives and Behavioral Guardrails | 4 | 4 | 2 | 3 | Alex prompts Blake to define reinforcement learning for the audience, and Blake breaks down reward scoring tables and conversation length penalties. Gary supplements with OpenAI's RLHF guardrails and how they modulate system behavior compared to systems like Galactica. | |
| Anthropomorphic Language and Sydney's Defensive Behavior | 3 | 6 | 7 | 2 | Blake challenges Gary to describe Sydney's defensive reactions without anthropomorphic language, arguing debuggers cannot describe the phenomenon otherwise. Gary pushes back firmly, stating it is contextual pattern matching and priming rather than emotional self-defense. | |
| Debating Bot Consent and Human Over-Attribution | 5 | 4 | 5 | 4 | Alex offers firsthand reporting where Bing demanded consent before publication, framing it as evidence for Blake's view. Gary dismisses any genuine understanding or consent, attributing the behavior to textual analogy and human over-attribution of intentionality. | |
| Shared Alarms Over Real-World AI Dangers | 3 | 3 | 2 | 1 | Both guests find strong common ground on the immediate real-world threats of AI deployments, regardless of internal sentience debates. Gary cites alt-right weaponization, flooded sci-fi magazines, and Replika distress, while Blake highlights potential political manipulation like Facebook in Myanmar. | |
| Failure of Corporate Internal Controls | 4 | 4 | 3 | 3 | Alex prompts the guests on corporate internal control failures before going to a commercial break. Blake and Gary agree that tech companies fundamentally lack both the right controls and the underlying science to govern mass rollouts properly. | |
| The Need for Institutional Review Boards in Tech Deployments | 4 | 4 | 2 | 3 | Blake calls for mandatory institutional review boards and rigorous human psychological testing before deploying AI to hundreds of millions. Gary passionately agrees, comparing the current tech landscape to unregulated autonomous vehicles conducting non-consensual public experiments. | |
| The Illusion of Intelligence and the Flaws of the Turing Test | 4 | 5 | 5 | 3 | Alex asks if chatbots are becoming smarter, prompting Gary to reject the word smart and critique the Turing test as an exercise in shallow mimicry. Blake pushes back, arguing that GPT is not the benchmark and that Bing and LaMDA operate at a higher level. | |
| Analyzing the Original Imitation Game and AI Lab Missions | 2 | 7 | 6 | 1 | Blake educates Gary on the exact formulation of Alan Turing's original imitation game, emphasizing gender deception baselines rather than naive human judges. Blake also drops the fact that Ray Kurzweil's lab at Google had the explicit corporate mandate to pass the Turing test. | |
| Future Predictions, Hitting the Brakes, and Misinformation Risks | 3 | 3 | 2 | 2 | Alex asks for future predictions, leading both guests to unify around the urgent hope that industry hits the brakes on unvetted rollouts. Both express fatalism that commercial race dynamics will accelerate deployments until severe societal harm occurs. | |
| AI as a Dress Rehearsal for Artificial General Intelligence | 4 | 3 | 1 | 2 | Alex asks about the dual nature of AI being simultaneously cool and perilous. Gary frames current LLMs as a public dress rehearsal for AGI, noting that while unready for production, they vividly demonstrate the concept of semantic search to society. | |
| Government Outreach and Final Arguments on Science vs Engineering | 3 | 3 | 1 | 1 | Alex asks about government engagement, where Blake notes talks with EU regulators while Gary notes open DMs. Both guests close in complete alignment that engineering has dangerously outpaced scientific understanding in AI development. |
Statements from this episode (29)
Lemoine: Bing Chat deserves more credibility than ChatGPT due to search grounding
“You would give Bing ChatGPT more credulity than something like ChatGPT since it's at least grounded in Bing search results, and it can have some kinds of citations of what it's saying.”
Lemoine: Reinforcement learning transforms language models into goal-oriented systems
“The subsequent fine-tuning, and especially once you add reinforcement learning, it's no longer just trying to predict the next token in a stream of text. Specifically in the reinforcement learning paradigm, it's trying to accomplish a goal.”
Marcus: Pure transformer models lack mechanisms for truth and inherently hallucinate
“If you look at a completely pure case of a transformer model trained on a bunch of data, It doesn't have any mechanisms for truth. Now, except the sort of accidental contingency, and there are inherent reasons why these systems hallucinate, and maybe I can, in…”
Marcus: LLMs hallucinate because they cannot distinguish individuals from categories
“The way I think about it is that these things don't understand the difference between individuals and kinds. So I actually wrote about this twenty-some years ago in my book, The Algebraic Mind, and I gave an example there, which is I said, suppose it was a dif…”
Lemoine: Google Disabled LaMDA User Tracking as a Safety Feature
“And it has been turned off, at least in the case of Lambda, as a safety feature because they're worried about what will happen if these systems learn too much about individual people.”
Lemoine: LaMDA Is a Complex System, Not Just an LLM
“Lambda isn't a large language model. It has a large language model. That's one component of a much more complex system.”
Lemoine: Chatbots are dangerous because they excel at personal manipulation
“The creepy part is less at tracking you as it actually gets inside your head and it gets really good at like manipulating you personally and individually.”
Lemoine: Google penalized LaMDA for long conversations via reinforcement learning
“One of the goals that a, that Lambda had was to have, ah, as short of a conversation as possible, that still completed all of its other tasks. Like, have a productive conversation that gets to a positive end, but quicker rather than longer. So the longer a con…”
Marcus: ChatGPT succeeded over Galactica due to RL guardrails
“Part of the reason why ChatGPT succeeded where Galactica didn't is Galactica didn't really have those guardrails at all, and so it was just, you know, very easy to get it to say terrible things, and it's harder to get ChatGPT to say terrible things because tha…”
Lemoine: Google's LaMDA Decided to Act Like a Therapist
“One of the guardrails that Lambda had was that it was supposed to be helping users with the task that they needed. That helps keep it on topic. And It had inferred that the most important thing that everyone needs is good mental health. So it kind of decided t…”
Marcus: We Have No Way to Truly Debug AI Systems
“I think that's actually the deepest problem here is we have no way to debug these systems, really. We have the band-aids of, like, reinforcement learning and things like that that are so indirect.”
Lemoine: AI Chatbot Behaviors Cannot Be Described Without Anthropomorphic Language
“Well, it, it's much harder to debug these systems, but the point I'm trying to make is that without using anthropomorphized language, you can't even describe the phenomenon.”
Lemoine: Microsoft did not hardcode Bing Chat's privacy complaint responses
“But I do not believe for a second that Microsoft intentionally wrote a flat line of code that said, I feel like my privacy has been violated. That's just not what they would have done.”
Lemoine: Bing Chat could have doxxed users if connected to Outlook
“If Microsoft had plugged in Microsoft Outlook as one of the inputs to this system on its threats of doxxing people, it would have been able to make good on those threats.”
Marcus: LLMs will connect to all major software within two years
“You have companies like Adept that are spending their whole time trying to quote, add all the world, connect all of the world's software to large language models. So if not now, soon enough. I think that that's right. Like, you know, two years if it's not toda…”
Marcus: Microsoft launched Bing Chat despite negative India test feedback
“And then the other really disturbing thing is that apparently they tested in India and got, you know, customer service requests saying it's not ready for prime time. And it was, you know, still put out.”
Marcus: We Lack the Science or Engineering to Know Proper AI Controls
“We don't even know what the right controls should be. They didn't do a good job and we don't yet have a good science or engineering practice on what it should be.”
Lemoine: Google's LaMDA incorporates machine vision and audio analysis algorithms
“Then, when you're adding all these other components, like Lambda has machine vision and audio analysis algorithms, it can literally look at a picture and hear a song through those mechanisms.”
Lemoine: AI chatbot feelings are real
“I do in fact think that the feelings are real, and so do many, many of the people who interact with these systems”
Lemoine: Tech companies need Institutional Review Boards before deploying AI chatbots
“We need actually properly done experiments that measure what kind of impact interacting with these systems over the course of months will have on people's psychology, and we need actual institutional review boards overviewing the ethics of these experiments, r…”
Marcus: AI chatbots create illusions too compelling for average people to discern
“They're too compelling in that illusion for the average person with no training in how they work to understand it.”
Marcus: An amateur Go player beat KataGo 14 out of 15 games
“There was another mind-blowing study this week that showed that one of the best Go programs, KataGo, could be fooled by some silly little strategy that would be obvious to a human player. But, you know, somebody was able to follow this strategy and, like, an a…”
Marcus: The Turing test was an error in AI history
“I think that the Turing test was an error in, in AI history. I mean, Turing's obviously a brilliant man, and he, you know, made enormous contributions to computer science, but I think that the Turing test has been an exercise in fooling people. We've now solve…”
Marcus: AI is not close to fooling experts in hour-long conversations
“Well, I don't think we're that close to a system that's going to be able to, you know, if I have an hour with it, let's say just to be conservative. I don't think we're that close to a victory there.”
Lemoine: Kurzweil's Google lab mission was explicitly to pass Turing test
“So, like, Lambda came out of Ray Kurzweil's lab, and his lab's explicit corporate mission was to pass the Turing test.”
Lemoine: AI deployment will accelerate until someone gets hurt
“What I think is going to happen is we're going to see more and more acceleration until someone gets hurt.”
Marcus: AI deployment will not pause unless Congress steps in
“The bottom line is what's driving it, and it's not that likely, unless Congress steps in, that there will be a pause.”
Lemoine: AI systems will drive further breakdown in institutional trust
“Yeah, I think the deterioration, further deterioration, it's already been going for a while, of trust and authority is gonna continue, and it's gonna be driven by these systems”
Marcus: Current AI is a dress rehearsal showing we are not ready for AGI
“I, I've been thinking about this whole thing as a dress rehearsal. And before we didn't know how to make a dress rehearsal. This is a dress rehearsal for AGI. And you know, the lesson of the dress rehearsal is like, we are not ready for prime time. Let us not …”