Windowed Transformer
topic on 1 show · 1 statements across 1 episodes
1 statements about Windowed Transformer, every show
Bachman: Models claiming 256k+ context use windowed transformers, discarding data
“Anybody who says they're using a transformer
With a context length of, you know, 256,000 or more, they're not using a true transformer.
What they're using is a windowed transformer that essentially throws out a huge amount of its information at various layers …”