Infinite Context Windows
topic on 3 shows · 3 statements across 3 episodes
Latent Space 
the Startup Ideas Podcast 
the a16z Podcast
3 statements about Infinite Context Windows, every show
Liu: AI models will never have infinite context windows due to physics
“I just don't think we're ever going to get to a point to where like a, an AI model can like have infinite context window. Right. And I think there's like a physics to that, right? Like you could just literally only have so much attention on like so much, you k…”
Crivello: Infinite and cheap context windows will arrive within 18 months
“Now we just assume that infinite context windows are going to be here in a year or something, a year and a half and infinitely cheap as well. And dynamic compute is going to be here. Like we just assume all of these things are going to happen.”
Amodei: AI models will never have infinite context windows due to compute
“Really the main thing holding back infinite context windows is just, you know, as you make the context window longer and longer, of course, the majority of the compute starts to be in the context window. So at some point it just becomes too, too expensive in t…”