Context Length

“The difference is if you compress the temporal dimension, you get a much higher compression rate. Because there is temporal redundancy between frames, because this frame and the last frame, likely they are mostly similar. So there's only some small difference.…”

“We took, you know, three years to go from 100 K context to one million context in every frontier model, but we're not going to a billion, you're not going to a trillion with context graphs, context lengths.”

“Really tied into this context thing, I think the context size will stay ballpark the same way it is, maybe a few times larger. But the parameter count will go up. Like, parameter count should grow much, much faster than context length, actually, just because o…”

“As you say that needed single needle in a haystack Benchmark is really saturated for at least context lengths up to one 28 K or something.”

“Extending the context length is kind of the easiest way to go about it, but there's quite a bit of progress that is being made on this.”

“And from a, from an, from a machine learning perspective, from an AI perspective, there's no reason we can't make the context length a hundred million words today, right? Which is roughly what a human hears in their lifetime.”

“I think we always expand our usage to fit in whatever context length. That's going to be available.”

“I expect, I mean, order of one, two years, you might see, wow, we went another order of magnitude from both state of the art and, of course, what's considered a commodity. So I'm pretty certain about this.”

“Well, it will, as a neural network, it will happily keep going on for infinite context, man. It will just keep generating. does it do well? That's the answer is no, because if you didn't train it to handle that situation, and that's actually a child rule. So,…”