Q of like chart or diagram. I think like I still, it's hard to understand when an LLM is sleeping. Like I understand that, but like if I were to use like, you know, look at the paper, it's like, is it, how do you define sleep time? Like when does sleep time start? When does it end? Is it on demand? Is it a innate part of the LLM inference?
A Yeah, I think like, we basically think that sleep time is like any time post-training that's not test time. I, I think that also kind of intuitively makes a lot of sense because like the LM, if it's running in a data center somewhere, if you have your GPU live, like you could be using it, like inferencing on it at any given point in time, even when the user is not actually, or like an event, an active event isn't happening. So it's basically like, yeah, it's like every time that is not test time, you know, where you could Presumably be doing some sort of compute. That's what we're calling sleep time. I think, you know, to go back to the analogy thing, I think, you know, with like MemGPT, we also thought a lot about like, you know, is the systems analogy better or is like cognitive analogy better? Um, I think the cognitive analogy for MemGPT is like, oh, well, a human has like different levels of like representations of their brain and like, you know, maybe the core memories is going to be something that's like more high level and then archival is like something more abstract. Um, I think similar to MemGPT, uh, I think the cognitive analogies are almost like a subset, kind of like Kevin was saying, of the systems level analogies. And I think the system level analogies are just a lot sharper because, you know, at the end of the day, like with tokens, it's like a memory hierarchy …
AI assessment note: “sleep time is like any time post-training that's not test time.”