Trillion Tokens
topic on 1 show · 1 statements across 1 episodes
1 statements about Trillion Tokens, every show
Dean: Scaling quadratic attention cannot reach billion- or trillion-token context windows
“But that's not going to be solved by purely scaling the existing solutions, which are quadratic. So a million tokens kind of pushes what you can do. You're not going to do that to a trillion tokens, let alone, you know, a billion tokens, let alone a trillion.”