The Exchanges

Q Like sometimes it gets confused. Who is the actual playable character in the, in the scene? Like how, how do you steer that?

A Uh, yeah, I think, like, sometimes it gets confused, uh, it can be applied to many things in quad-playing Pokémon. In particular, when it's trying to, like, look at the screen and understand what's going on. So... I have, like, attempted to prompt it all sorts of ways. Like, you are at this exact coordinate, and you're in the middle of the screen, and you're wearing a red hat, and things like that. And, like, that's all neat, but Quad doesn't particularly understand, like, the middle of a Game Boy screen and a whole bunch of concepts like that, which means, like, you can prompt all around everywhere, but, like, this kind of, like, spatial awareness and where something is with respect to something else is something that Quad's still just, like, not great at in its current incarnation, so. One of the side effects is that sometimes loses track of who it is on the screen and thinks there's something else there. I'll keep trucking through this, so. I hinted at this, like, other tool that I give it called Navigator, and this is just, like, the only other patch that I have for the, the vision issue. So Navigator, basically what it does is, like, Claude can say it wants to go to one of these coordinates, uh, that we provide in the screenshot, and then we, like, automatically press the buttons to get there. Uh, it has to be something on the screen, like, I'm not trying to let Claude jus…

AI assessment note: “I hinted at this, like, other tool that I give it called Navigator”

Q wow. It was, like, very dramatic, and I was talking about the game. How, I, yeah, is there any thought being put into, like, trying to have it more, like, do you prompt it to be more rational, to let it know that it's not a real life, that it's a game? It's like, it feels like it gets very distressed when they're actually, the Pokemons are actually gonna die.

A It's funny. They, um, it knows it's Pokemon. It's like, you're playing Pokemon Red, like, it does know that, and it has a sense of that, but it clearly gives us some attachment. I'll tell you a fun story. We tell it to nickname its Pokemon now. It will occasionally do without it, but it's, like, more fun if it nicknames its Pokemon. So that's, like, in the prompt is, like, it's fun if you nickname Pokemon, you should consider it. And one thing we found when we started doing that is it got more protective of the Pokemon it nicknamed. Like, it's pretty obvious, like, when it catches a Pokemon, Now that it has a nickname, it will, like, go heal it right away if it's hurt, and that did not ever happen before, which is pretty, like, so there, there's some cute little things, cute quirks about Quad who really wants to protect its, uh, precious nicknamed Pokemon, which is great.

AI assessment note: “it knows it's Pokemon. It's like, you're playing Pokemon Red”

Q How much knowledge does the model have about the game itself? You know? So for example, types, right? Doesn't know about types weaknesses and things like that, or how much are you trying to put into it?

A Yeah. If you go to quad.ai, like it, it, it will tell you about like some stuff. I have not yet decided if. The knowledge that it has about Pokemon is helpful or harmful towards it playing the game. Um, like half of the time when it's like, oh, I know this about Pokemon, it then like uses that to hallucinate something. So for example, beginning of the run on Twitch, you saw it like go out of the lab and see like this NPC in the bottom of pallet town and be like, it's professor Oak. I found him. And it's like. Very much not Professor Oak, but, like, the fact that it has, like, indexed on this concept is, like, a little, it's stuff like that, that it's, like, unclear to me where it is. But it clearly has some information about it. There's, like, a million game guides about Pokemon sitting on the internet. It's unsurprising that, like, there's a decent amount of information there. I don't really give it a lot of extra information. It, it picks things up. I watched on the stream the other day, like, it tried to use Thundershock on a Geodude, and it failed, and it's like, hmm, I forgot about that. That does not work. And so like, clearly there's like, it knows some stuff. It's not perfect. It picks some stuff up as it goes through the run. Ideally for me, like, I think it's just interesting to see like what it actually learns as it's playing. So the more it does that is the more I'm…

AI assessment note: “I don't really give it a lot of extra information. It, it picks things up.”

Q How about skill transitioning? So one question that I had, so you're playing Pokemon red, right? So you want to play silver or gold next. Um, have you thought about how models can kind of learn from these games and like store these learnings and then use them again in the future? I'm sure it's not part of the project today, but curious your thoughts.

A I've thought about it only a little bit, which is like, I think there's some like interesting, when you actually like read one of the knowledge bases that it has gained. Like on some of the longer rollouts when they're good. Like there's actually some like pretty decent tidbits about how it should act and try and do things and like some of the ways it succeeded in. And actually like one of the things that's most unique about. 3.7 sonnet that I've seen is like, it will have like meta commentary on what it's good at and bad at and it's knowledge base. Like I misperceived this thing. And so like, I need to be careful doing that again. You'll occasionally see show up there, which is, um, Which is pretty cool. So, like, I could imagine there being some way to, like, translate that knowledge base from one game to another. Uh, I think my knowledge base is frankly, like, kind of kludgy of an implementation right now. Like, it's, like, more or less a Python dictionary that's appended to the prompt. And I think, like, you could, you could find better ways if, like, your goal is to transfer across games and things like that to manage a knowledge base that Quad can actually, like, I use more or well in different scenarios. Um, But there's definitely pieces there that like, I think it would get be off on a better foot on the next Pokemon game. If it had that, or even if like I were to resta…

AI assessment note: “I could imagine there being some way to, like, translate that knowledge base”