Huang: Context scaling requires positional interpolation rather than extrapolation
Mark Huang · How to train a Million Context LLM — with Mark Huang of Gradient.ai · May 31, 2024 · at 21:04
Mark Huang, co-founder of Gradient.ai, discusses technical mechanisms of RoPE scaling and positional embeddings for expanding LLM context windows.
0:00 / 0:36exact quote · 36.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“There's it's super confusing, but it's like, there's extrapolate positional extrapolation, and then there's interpolation. You want interpolation. It's been shown that just pure extrapolation makes the model a lot worse, and it's harder to attend to stuff, whereas the interpolation is like, you're squeezing everything back in to what the original context length was to a certain extent, and then allowing for it to overlap Different sequences that it's already seen as if it actually occurred when you see a billion contexts of sequence tokens.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Mark Huang
Huang: True AI agents require measurable probability improvements per node
“It's like on each stage of the node, you're gonna have to see a marginal improvement in the probability of success for that particular workload because of non-determinism.”
Huang: Google's internal AI tooling was far superior to competitors
“Google was using AI for systems before everybody else too, right? They invented a transformer, and their internal set of tooling was just so far superior to everything else. Like, it's really hard for people to go back after seeing that.”
Huang: RAG versus fine-tuning is fundamentally just meta-learning
“And like, at the end of the day, it's just all meta-learning, right? Like, all we want is, like, the best meta learning workflow or meta learning setup possible to be able to adapt the model to do anything.”
Huang: PoSE breaks down on needle-in-a-haystack at 500k tokens
“It does start to break down a little bit more on the longer, longer context. So, like, 500,000 to a million it appeared that it doesn't hold as well specifically for, like, needle in the haystack.”
Huang: Adding one billion tokens cannot teach trillion-token models new knowledge
“All models these days are now double-digit trillions, right? So it's kind of a drop in the bucket if you really think I can just put, you know, a billion tokens in there, and I actually think that the model's gonna truly learn new Information.”