Mar 4, 2025 · 37m · latent-space
How Claude Plays Pokémon was made
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Anthropic engineer David Hershey joins Latent Space to detail how he built 'Claude Plays Pokémon,' examining agent architecture, spatial reasoning hurdles, context window management, and the evolution of autonomous capabilities in Claude 3.7 Sonnet.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 14.8% of the talking time here. How this is scored →
The hosts as informed peer 4.0 Guest teaching 4.9 Guest disagreement 1.3 The hosts pushing back 1.1
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 30:00 David dismisses prompt engineering as a cure for navigation
David politely but firmly rejects the Twitch community's assertions that clever prompt crafting can fix Claude's spatial disorientation.
Hardest push from the hosts ▶ 20:42 Alessio queries Claude live on the Mount Moon walkthroughAlessio directly tests Claude's pathing logic live on Claude.ai to challenge whether navigation failure stems from missing game knowledge.
Biggest teaching moment ▶ 22:30 David explains the counter-intuitive power of prompt deletionDavid educates the hosts on how human intuition and prescriptive prompting actually degraded performance relative to giving raw reasoning models free reign.
The host holds their own ▶ 27:29 Alessio connects game learning to TCG tempo transferAlessio demonstrates domain depth by comparing agent cross-game knowledge transfer to universal tempo concepts across Magic and Flesh and Blood.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Origins and Evolution of Claude Plays Pokémon | 4 | 3 | 1 | 1 | Alessio sets up the premise comparing Claude Plays Pokémon to Twitch Plays Pokémon and asks about game mechanics like isometric views. David details the project's progression across Claude model versions from 3.5 Sonnet to 3.7 Sonnet. | |
| System Architecture and Emulator Integration | 4 | 5 | 1 | 1 | David walks through the harness architecture diagram, explaining tool definitions, RAM reverse-engineering, and hallucination workarounds. Vibu and Alessio ask technical clarifying questions about Game Boy coordinates and emulator state extraction. | |
| Spatial Vision Challenges and the Navigator Tool | 3 | 6 | 1 | 1 | Alessio asks how much domain knowledge the model has and how Claude handles spatial self-awareness. David educates on how Claude's visual reasoning struggles with 2D Game Boy sprites, requiring the custom Navigator tool patch. | |
| Token Economics and Context Window Management | 4 | 5 | 1 | 1 | Vibu probes into context window mechanics, token economics, and truncation boundaries. David details the exact breakdown of token budgets, 30-step rollout cycles, and summarization trigger points. | |
| Model Reasoning and Prompt Simplification | 4 | 6 | 2 | 2 | Alessio checks Claude's pathfinding plan for Mount Moon, and Vibu questions whether pure guide memorization is ideal. David explains the counterintuitive discovery that removing prompt band-aids yielded better reasoning performance on Claude 3.7. | |
| Emergent Behaviors and Cross-Game Skill Transfer | 5 | 4 | 1 | 1 | Alessio draws an analogy to cross-game conceptual learning in Magic: The Gathering and trading card games. David agrees and explains how the model writes its own meta-commentary inside the prompt dictionary to learn from tactical mistakes. | |
| Overcoming Game Bottlenecks and Milestone Highlights | 4 | 5 | 2 | 1 | Vibu inquires about community optimization opportunities and formal evaluation metrics. David pushes back on the idea that prompt tweaks can fix fundamental vision gaps, recounting Claude repeatedly entering and exiting Oak's lab in an infinite loop. |
Statements from this episode (16)
Pokémon is ideal for testing AI agents because delays bring no penalty
“Pokemon's actually really nice because, like, if you don't do anything for five seconds, like, there's typically not a consequence by the nature of, like, doing inference on a model every, like, snapshot of time. It's actually a pretty good game to be able to …”
Claude aggressively hallucinates game zone transitions without explicit negative feedback
“Claude will, like, pretty aggressively hallucinate that it succeeded in transitioning between zones if you don't, like, tell it did not.”
Claude spent 12 hours overnight mistaking a Pokémon doormat for a textbox
“I once saw it see like a red box on the screen that was like the doormat and think it was a text box and spend 12 hours pressing A overnight to try to clear the text box, which you see that happen once and you add in some helpful reminders to not do that.”
Claude still struggles with spatial awareness and visual positioning on screen
“Quad doesn't particularly understand, like, the middle of a Game Boy screen and a whole bunch of concepts like that, which means, like, you can prompt all around everywhere, but, like, this kind of, like, spatial awareness and where something is with respect t…”
Claude Plays Pokémon API requests max out around 100,000 tokens
“So in practice, this rollout ends up, like, at max, ending up around a 100,000 tokens, I think, is where it is, like, the longest message you ever send to the API on one of these turns, and it will fluctuate in, like, summarization, depending on the state of k…”
Running Claude Plays Pokémon experiments costs thousands of dollars in API tokens
“There's like at least thousands of dollars of tokens being consumed. So it's not a, it is not a cheap rollout.”
AI agents have an optimal effective context length where intelligence peaks
“I think like one thing you see a lot when you talk to people building agents is there's like some effective context length that actually like has the model be the smartest. And that seems to vary slightly model by model, but for this model, for whatever purpos…”
Prompting Claude cannot improve its spatial navigation without explicit instructions
“You can try to prompt Quad a lot of different ways to understand how to navigate better, and anything short of telling it exactly what to do does not improve its, like, actual navigation.”
Upgrading Claude models in Pokémon primarily involves deleting prompt scaffolding
“Literally every model that has come out with Pokemon, like, the main change that I have made to this agent is deleting prompt stuff.”
Prompting Claude to nickname its Pokémon caused it to exhibit protective behaviors
“And one thing we found when we started doing that is it got more protective of the Pokemon it nicknamed. Like, it's pretty obvious, like, when it catches a Pokemon, Now that it has a nickname, it will, like, go heal it right away if it's hurt, and that did not…”
Claude 3.7 Sonnet uniquely generates meta-commentary reflecting on its perceptual errors
“And actually like one of the things that's most unique about. 3.7 sonnet that I've seen is like, it will have like meta commentary on what it's good at and bad at and it's knowledge base. Like I misperceived this thing. And so like, I need to be careful doing …”
The Claude Plays Pokémon knowledge base is simply a prompted Python dictionary
“I think my knowledge base is frankly, like, kind of kludgy of an implementation right now. Like, it's, like, more or less a Python dictionary that's appended to the prompt.”
Hershey predicts the Claude stream won't reach Victory Road within 16 days
“I think we have a little ways before we can beat the game in 16 days. I do not have a lot of faith that the current stream is gonna, gonna be standing in Victory Road in 13 days.”
Anthropic's Pokémon research graph reflects a single run passing Lt. Surge
“The run that you saw that's, like, on the graph we put out alongside, like, in our research blog is, like, a single run that I have watched, like, get through At least surges Jim. And then it got a little past that. And the reason that that's where we stopped …”
Claude 3.7's error-correction capabilities will enable better real-world AI agents
“I really do think like this is just demonstrating like a thing that is going to make agents better with this model, you know, like This is a very fun way to see it, but, like, I think the thing is that it, like, has some ability to, like, course correct, updat…”