Quadratic Attention
topic on 1 show · 2 statements across 2 episodes
2 statements about Quadratic Attention, every show
Dean: Scaling quadratic attention cannot reach billion- or trillion-token context windows
“But that's not going to be solved by purely scaling the existing solutions, which are quadratic. So a million tokens kind of pushes what you can do. You're not going to do that to a trillion tokens, let alone, you know, a billion tokens, let alone a trillion.”