Sleep Time
topic on 1 show · 2 statements across 1 episodes
2 statements about Sleep Time, every show
Packer: Sleep-time compute avoids the user latency costs of test-time compute
“Assigning higher costs to tokens that come at test time, because once you're at test time, it kind of implies that a user is waiting. Something is waiting. It's either another process, a user, an event, and there is real cost to like every single token or like…”
Packer: Sleep-time compute during idle downtime is a major missed opportunity
“And practically speaking, you know, machines, they're not like humans, they can be run all the time. And there's a ton of downtime, both in advance of like questions being asked also like After questions have been asked too. So I think beyond just scaling at t…”