Claude 3.7's error-correction capabilities will enable better real-world AI agents
David Hershey · How Claude Plays Pokémon was made · Mar 4, 2025 · at 35:39
David Hershey, an Anthropic engineer and creator of Claude Plays Pokémon, explains why the agentic capabilities demonstrated in the Pokémon stream translate to real-world AI agents.
0:00 / 0:37exact quote · 37.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I really do think like this is just demonstrating like a thing that is going to make agents better with this model, you know, like This is a very fun way to see it, but, like, I think the thing is that it, like, has some ability to, like, course correct, update, and figure things out a little bit better than models have in the past, and even if there's, like, stuff it's dumb at, like, it tends to have an ability to, like, power through it in a new way, and so I think what it's exciting to me is just, like, I think there will be some real-world stuff that comes out of this model once people play with it, and I'm pretty excited to see, like, how people take the skills we put on display a little bit here, or lack thereof in some cases, and figure out how to turn them into Actual agents that do stuff.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from David Hershey
Hershey: Claude 3.7 extended thinking does not help Pokémon gameplay
“I've tested, like, all sorts of the extended thinking mode with, ah, 3.7 on it, and, like, it doesn't really help.”
Hershey: Anthropic Study Found Claude Treats Named Characters Better
“Anthropic actually did like a blinded study of like named characters versus unnamed characters in different settings, and Claude like actually does clearly prefer and is nicer to named characters, which is an interesting thing.”
AI agents have an optimal effective context length where intelligence peaks
“I think like one thing you see a lot when you talk to people building agents is there's like some effective context length that actually like has the model be the smartest. And that seems to vary slightly model by model, but for this model, for whatever purpos…”
Upgrading Claude models in Pokémon primarily involves deleting prompt scaffolding
“Literally every model that has come out with Pokemon, like, the main change that I have made to this agent is deleting prompt stuff.”
Prompting Claude to nickname its Pokémon caused it to exhibit protective behaviors
“And one thing we found when we started doing that is it got more protective of the Pokemon it nicknamed. Like, it's pretty obvious, like, when it catches a Pokemon, Now that it has a nickname, it will, like, go heal it right away if it's hurt, and that did not…”
Claude 3.7 Sonnet uniquely generates meta-commentary reflecting on its perceptual errors
“And actually like one of the things that's most unique about. 3.7 sonnet that I've seen is like, it will have like meta commentary on what it's good at and bad at and it's knowledge base. Like I misperceived this thing. And so like, I need to be careful doing …”