Power Retention

“We're going to be completely open sourcing all of the pieces that you need to do this metamorphosis yourself”

“After just 10,000 steps of training, which this training one took about two hours, this orange curve, you see that it fully matches the original loss.”

“They'll come out with a nice shiny new, a power retention architecture that has the same performance on whatever data set they want as the original base model did.”

“Power retention is the specific variant that we're about to release. And it basically works by instead of the memory constantly growing, this constantly Growing KVCache. You have a memory that is a fixed size and each new token simply gets compressed into this…”

“So yeah, but we don't pay a quadratic cost. If you were looking at the star coder baseline, it would get even more, more expensive way more quickly.”

“Actually, our initial version of power retention was written in Triton, but we realized quickly that it just didn't offer the flexibility to really squeeze the performance that we wanted out of the GPU. So we took a step back and dove into CUDA.”

“And at 64 K tokens, We get something like a 10 X speed up at training, but at inference time, because you're not only saving flops at inference time, but also paging in and out of memory of the KV cache, you actually get a hundred X speed ups from power retent…”

“After that, I think, you know, probably within six months to a year, we're going to start to see the really big foundation models being trained in this way.”

“That's something we plan on doing at the like, thirty billion scale in the coming months.”