Pokémon

“The thing that Claude's the worst at is understanding how to get from point A to point B. It's, like, really, really god-awful at hitting the buttons to go from point A to point B on a screen.”

“Building evals that actually test, like, time performance over, like, days of token sampling are quite hard. Like, it's actually really, really hard to build things that you can measure In a reasonable way. And one, this is one of them. Like, it's an eval that…”

“Sonnet three point oh, I eventually hooked up to this, just because, like, fun to see what would happen. And it started role playing its own progress of the game, so like, it stopped being able to make progress, and instead it said, so like, now I'm going to g…”

“People who have watched the stream often complain about the fact that I don't let Claude press multiple buttons when there's dialogue on the screen anymore. That's because I watched a run, or a run, where it had an Ivysaur in Mt. Moon, and it had Tackle as its…”

“You can try to prompt Quad a lot of different ways to understand how to navigate better, and anything short of telling it exactly what to do does not improve its, like, actual navigation.”

“Literally every model that has come out with Pokemon, like, the main change that I have made to this agent is deleting prompt stuff.”

“And one thing we found when we started doing that is it got more protective of the Pokemon it nicknamed. Like, it's pretty obvious, like, when it catches a Pokemon, Now that it has a nickname, it will, like, go heal it right away if it's hurt, and that did not…”