quadratic attention
2 statements across 2 episodes · 0 bullish · 2 bearish · 2 people on the record · first statement Dec 24, 2024 by Dan Fu · across every show →
Everything said about quadratic attention, oldest first
Dec 24, 2024 bearish
Feb 12, 2026 bearish
Dean: Scaling quadratic attention cannot reach billion- or trillion-token context windows
“But that's not going to be solved by purely scaling the existing solutions, which are quadratic. So a million tokens kind of pushes what you can do. You're not going to do that to a trillion tokens, let alone, you know, a billion tokens, let alone a trillion.”