The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 27 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Hershey: Claude 3.7 extended thinking does not help Pokémon gameplay
“I've tested, like, all sorts of the extended thinking mode with, ah, 3.7 on it, and, like, it doesn't really help.”
Hershey: Anthropic Study Found Claude Treats Named Characters Better
“Anthropic actually did like a blinded study of like named characters versus unnamed characters in different settings, and Claude like actually does clearly prefer and is nicer to named characters, which is an interesting thing.”
AI agents have an optimal effective context length where intelligence peaks
“I think like one thing you see a lot when you talk to people building agents is there's like some effective context length that actually like has the model be the smartest. And that seems to vary slightly model by model, but for this model, for whatever purpos…”
Upgrading Claude models in Pokémon primarily involves deleting prompt scaffolding
“Literally every model that has come out with Pokemon, like, the main change that I have made to this agent is deleting prompt stuff.”
Prompting Claude to nickname its Pokémon caused it to exhibit protective behaviors
“And one thing we found when we started doing that is it got more protective of the Pokemon it nicknamed. Like, it's pretty obvious, like, when it catches a Pokemon, Now that it has a nickname, it will, like, go heal it right away if it's hurt, and that did not…”
Claude 3.7 Sonnet uniquely generates meta-commentary reflecting on its perceptual errors
“And actually like one of the things that's most unique about. 3.7 sonnet that I've seen is like, it will have like meta commentary on what it's good at and bad at and it's knowledge base. Like I misperceived this thing. And so like, I need to be careful doing …”
Hershey: Claude is exceptionally bad at point-to-point spatial navigation
“The thing that Claude's the worst at is understanding how to get from point A to point B. It's, like, really, really god-awful at hitting the buttons to go from point A to point B on a screen.”
Hershey: Pokémon serves as an effective multi-day evaluation benchmark for AI models
“Building evals that actually test, like, time performance over, like, days of token sampling are quite hard. Like, it's actually really, really hard to build things that you can measure In a reasonable way. And one, this is one of them. Like, it's an eval that…”
Hershey: Newer Claude Models Tenaciously Keep Trying Rather Than Quitting
“This is, like, maybe the thing that is the best about the new models is they have a tendency to, like, tenaciously still try things.”
Hershey: Prompt engineering cannot fix Claude's visual comprehension limitations
“Vision is, like, pretty beyond fixing with a prompt. Again, go for it. Have fun. I've spent a lot of hours, like, overlaying grids, overlaying images, stretching, compressing, contrast colors, all sorts of stuff.”
Hershey: Chain-of-thought between tool calls only improves agent progress 10%
“Forcing it to think between tool calls. It does help, but not much. It's, like, 10% faster progress if you, like, have it use a lot of chain of thought tokens every time it chooses an action.”
Hershey: Eight Historical Images Yields Peak Performance for Game Agents
“At least in my harness, something like eight historical images is the best performance. More than that, and you start flirting with the space where, like, you get drop off in performance from, like, just having more tokens in the context, which is a thing that…”
Hershey: Stacking Negative Scaffolding Instructions Degrades Agent Performance
“If you keep adding that type of instructions, you eventually make the model stupider. Like, if you add 50 of that type of instruction, you just end up, like, with the whole agent doing worse”
Claude spent 12 hours overnight mistaking a Pokémon doormat for a textbox
“I once saw it see like a red box on the screen that was like the doormat and think it was a text box and spend 12 hours pressing A overnight to try to clear the text box, which you see that happen once and you add in some helpful reminders to not do that.”
Claude still struggles with spatial awareness and visual positioning on screen
“Quad doesn't particularly understand, like, the middle of a Game Boy screen and a whole bunch of concepts like that, which means, like, you can prompt all around everywhere, but, like, this kind of, like, spatial awareness and where something is with respect t…”
Prompting Claude cannot improve its spatial navigation without explicit instructions
“You can try to prompt Quad a lot of different ways to understand how to navigate better, and anything short of telling it exactly what to do does not improve its, like, actual navigation.”
Hershey predicts the Claude stream won't reach Victory Road within 16 days
“I think we have a little ways before we can beat the game in 16 days. I do not have a lot of faith that the current stream is gonna, gonna be standing in Victory Road in 13 days.”
Claude 3.7's error-correction capabilities will enable better real-world AI agents
“I really do think like this is just demonstrating like a thing that is going to make agents better with this model, you know, like This is a very fun way to see it, but, like, I think the thing is that it, like, has some ability to, like, course correct, updat…”
Hershey: Claude 3.0 Sonnet Hallucinated Pokémon Progress Instead of Playing
“Sonnet three point oh, I eventually hooked up to this, just because, like, fun to see what would happen. And it started role playing its own progress of the game, so like, it stopped being able to make progress, and instead it said, so like, now I'm going to g…”
Pokémon is ideal for testing AI agents because delays bring no penalty
“Pokemon's actually really nice because, like, if you don't do anything for five seconds, like, there's typically not a consequence by the nature of, like, doing inference on a model every, like, snapshot of time. It's actually a pretty good game to be able to …”
Claude aggressively hallucinates game zone transitions without explicit negative feedback
“Claude will, like, pretty aggressively hallucinate that it succeeded in transitioning between zones if you don't, like, tell it did not.”
Claude Plays Pokémon API requests max out around 100,000 tokens
“So in practice, this rollout ends up, like, at max, ending up around a 100,000 tokens, I think, is where it is, like, the longest message you ever send to the API on one of these turns, and it will fluctuate in, like, summarization, depending on the state of k…”
Running Claude Plays Pokémon experiments costs thousands of dollars in API tokens
“There's like at least thousands of dollars of tokens being consumed. So it's not a, it is not a cheap rollout.”
The Claude Plays Pokémon knowledge base is simply a prompted Python dictionary
“I think my knowledge base is frankly, like, kind of kludgy of an implementation right now. Like, it's, like, more or less a Python dictionary that's appended to the prompt.”
Anthropic's Pokémon research graph reflects a single run passing Lt. Surge
“The run that you saw that's, like, on the graph we put out alongside, like, in our research blog is, like, a single run that I have watched, like, get through At least surges Jim. And then it got a little past that. And the reason that that's where we stopped …”
Measuring AI game progress is an integration test, not a unit test
“And so I think like how quickly it's able to make progress is actually a pretty reason or a reasonable like eval if a slightly expensive one to calculate. It's an integration test, not a unit test.”
Hershey Restricts Multi-Button Inputs After Claude Overwrote Ivysaur's Attack
“People who have watched the stream often complain about the fact that I don't let Claude press multiple buttons when there's dialogue on the screen anymore. That's because I watched a run, or a run, where it had an Ivysaur in Mt. Moon, and it had Tackle as its…”