positional interpolation
1 statements across 1 episodes · 1 bullish · 0 bearish · 1 people on the record · first statement May 31, 2024 by Mark Huang · across every show →
Everything said about positional interpolation, oldest first
May 31, 2024 positive
Huang: Context scaling requires positional interpolation rather than extrapolation
“There's it's super confusing, but it's like, there's extrapolate positional extrapolation, and then there's interpolation.
You want interpolation.
It's been shown that just pure extrapolation makes the model a lot worse, and it's harder to attend to stuff, whe…”