Apr 5, 2025 · 1h 14m · latent-space
Claude Plays Pokémon Hackathon: Escape from Mt. Moon!
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
This hackathon kickoff event explores the development, engineering challenges, and cloud infrastructure behind autonomous AI agents playing Pokémon, featuring technical talks from Anthropic, Mangrove Technology, and Morph Labs for the 'Escape from Mt. Moon' challenge.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 0.4 Guest teaching 0.6 Guest disagreement 0.3 The hosts pushing back 0.3
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 21:15 David doubts upscaling will solve spatial reasoning
David gently dismisses the host's idea of upscaling the screen with an auxiliary model, explaining that Claude's inability to calculate paths from A to B persists regardless of visual fidelity.
Hardest push from the hosts ▶ 21:05 Host proposes model-based image upscalingThe host pushes back against the premise that Game Boy vision is a permanent blocker by pitching an architecture that upscales game frames to realistic images.
Biggest teaching moment ▶ 21:15 David explains the core limitation is spatial logicDavid clarifies to the host that resolving screen clarity will not fix Claude's fundamental deficit in understanding relative positioning and navigation.
The host holds their own ▶ 21:05 Host pitches computer vision preprocessingThe host demonstrates technical problem-solving by proposing a multi-model pipeline where an upscaler bridges the visual gap for the primary agent.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| History and Evolution of Claude Plays Pokémon | 0 | 0 | 0 | 0 | David Hershey gives a solo talk detailing the timeline of building Claude Plays Pokémon across different Sonnet versions. The host does not engage during this presentation segment. | |
| Anecdotes and Funny Failure Modes of Claude | 0 | 0 | 0 | 0 | David continues his solo monologue recounting humorous failure modes of earlier Claude models in Pokémon, including reset requests and hallucinated roleplay. There is no host involvement. | |
| Model Limitations, Spatial Reasoning, and Prompting Tips | 0 | 0 | 0 | 0 | David presents on Claude's struggles with spatial reasoning, screen comprehension, and agent prompting strategies. No host interaction takes place. | |
| Q&A on Agent Prompting, Vision, and Architecture | 3 | 4 | 2 | 2 | The host and audience ask questions regarding vision and memory architectures. The host suggests using an upscaler model on low-res screens, but David explains why spatial reasoning remains the true bottleneck. | |
| Building an Autonomous Virtual Streamer with Ruby and RAM Reading | 0 | 0 | 0 | 0 | Andrew presents his virtual Pokémon streamer architecture using Ruby, RetroArch, and RAM reading. The host is not involved aside from minor setup logistics. | |
| Streamer Q&A and Hackathon Competition Rules | 0 | 0 | 0 | 0 | Andrew answers audience questions before the hackathon organizer outlines rules against hardcoded hints. The host does not participate substantively. | |
| Morph Cloud, Infinibranch Architecture, and the EVA Framework | 0 | 0 | 0 | 0 | Jesse Han explains Morph Cloud's InfiniBranch snapshot technology and the EVA verification framework for the hackathon. The host does not engage during this talk. |
Statements from this episode (11)
Hershey: Claude is exceptionally bad at point-to-point spatial navigation
“The thing that Claude's the worst at is understanding how to get from point A to point B. It's, like, really, really god-awful at hitting the buttons to go from point A to point B on a screen.”
Hershey: Pokémon serves as an effective multi-day evaluation benchmark for AI models
“Building evals that actually test, like, time performance over, like, days of token sampling are quite hard. Like, it's actually really, really hard to build things that you can measure In a reasonable way. And one, this is one of them. Like, it's an eval that…”
Hershey: Newer Claude Models Tenaciously Keep Trying Rather Than Quitting
“This is, like, maybe the thing that is the best about the new models is they have a tendency to, like, tenaciously still try things.”
Hershey: Claude 3.0 Sonnet Hallucinated Pokémon Progress Instead of Playing
“Sonnet three point oh, I eventually hooked up to this, just because, like, fun to see what would happen. And it started role playing its own progress of the game, so like, it stopped being able to make progress, and instead it said, so like, now I'm going to g…”
Hershey Restricts Multi-Button Inputs After Claude Overwrote Ivysaur's Attack
“People who have watched the stream often complain about the fact that I don't let Claude press multiple buttons when there's dialogue on the screen anymore. That's because I watched a run, or a run, where it had an Ivysaur in Mt. Moon, and it had Tackle as its…”
Hershey: Prompt engineering cannot fix Claude's visual comprehension limitations
“Vision is, like, pretty beyond fixing with a prompt. Again, go for it. Have fun. I've spent a lot of hours, like, overlaying grids, overlaying images, stretching, compressing, contrast colors, all sorts of stuff.”
Hershey: Claude 3.7 extended thinking does not help Pokémon gameplay
“I've tested, like, all sorts of the extended thinking mode with, ah, 3.7 on it, and, like, it doesn't really help.”
Hershey: Chain-of-thought between tool calls only improves agent progress 10%
“Forcing it to think between tool calls. It does help, but not much. It's, like, 10% faster progress if you, like, have it use a lot of chain of thought tokens every time it chooses an action.”
Hershey: Anthropic Study Found Claude Treats Named Characters Better
“Anthropic actually did like a blinded study of like named characters versus unnamed characters in different settings, and Claude like actually does clearly prefer and is nicer to named characters, which is an interesting thing.”
Hershey: Eight Historical Images Yields Peak Performance for Game Agents
“At least in my harness, something like eight historical images is the best performance. More than that, and you start flirting with the space where, like, you get drop off in performance from, like, just having more tokens in the context, which is a thing that…”