windowed transformer
1 statements across 1 episodes · 0 bullish · 1 bearish · 1 people on the record · first statement Sep 23, 2025 by Diego Bachman · across every show →
Everything said about windowed transformer, oldest first
Sep 23, 2025 negative
Bachman: Models claiming 256k+ context use windowed transformers, discarding data
“Anybody who says they're using a transformer
With a context length of, you know, 256,000 or more, they're not using a true transformer.
What they're using is a windowed transformer that essentially throws out a huge amount of its information at various layers …”