Bachman: Models claiming 256k+ context use windowed transformers, discarding data
Diego Bachman · ⚡️ Beyond Transformers with Power Retention · Sep 23, 2025 · at 2:58
Diego Bachman, co-founder of Manifest AI, discusses architectural shortcuts and context-length scaling in large language models.
0:00 / 0:16exact quote · 16.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Anybody who says they're using a transformer With a context length of, you know, 256,000 or more, they're not using a true transformer. What they're using is a windowed transformer that essentially throws out a huge amount of its information at various layers within the network.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Diego Bachman
Bachman: Power Retention Delivers 100x Inference Speedup at 64k Context
“And at 64 K tokens, We get something like a 10 X speed up at training, but at inference time, because you're not only saving flops at inference time, but also paging in and out of memory of the KV cache, you actually get a hundred X speed ups from power retent…”
Bachman: Power Retention models match original base model performance
“They'll come out with a nice shiny new, a power retention architecture that has the same performance on whatever data set they want as the original base model did.”
Bachman: StarCoder-3B converted to Power Retention matches baseline loss in two hours
“After just 10,000 steps of training, which this training one took about two hours, this orange curve, you see that it fully matches the original loss.”
Bachman: Big Foundation Models Will Train on Power Retention Within a Year
“After that, I think, you know, probably within six months to a year, we're going to start to see the really big foundation models being trained in this way.”
Bachman: Compute-optimal models on internet text don't need long context
“In general, most internet text has mostly short-term structure. There's just not that much value in capturing long-term structure, and so compute optimal models on internet text actually don't have that long context, and so, of course, you're perfectly fine us…”
Bachman: Manifest AI is releasing Power Retention architecture with fixed-size memory
“Power retention is the specific variant that we're about to release.
And it basically works by instead of the memory constantly growing, this constantly
Growing KVCache.
You have a memory that is a fixed size and each new token simply gets compressed into this…”