Positional Interpolation
topic on 1 show · 1 statements across 1 episodes
1 statements about Positional Interpolation, every show
Huang: Context scaling requires positional interpolation rather than extrapolation
“There's it's super confusing, but it's like, there's extrapolate positional extrapolation, and then there's interpolation.
You want interpolation.
It's been shown that just pure extrapolation makes the model a lot worse, and it's harder to attend to stuff, whe…”