LLM Context Window
topic on 3 shows · 4 statements across 3 episodes
Latent Space 
No Priors 
the a16z Podcast
4 statements about LLM Context Window, every show
Kirkos: Dumping 100,000 rows into LLMs degrades quality and inflates costs
“We tried to give it the whole sheet as the context initially, and it works really great when you're working on a really small sheet. And then if you know, drag and drop a 100,000 rows of data in, first of all, it's going to be really expensive. Your request is…”
Liberty: Context Window Stuffing Increases LLM Costs Without Improving Results
“There's plenty of evidence that increasing the context size doesn't actually improve results unless, you know, you do this very carefully, right? So just what's called constant stuffing is not helping. You just pay more and don't actually get much for it.”
Park: Expanding LLM context windows cannot replace external agent memory
“And even if that limitation were to go away in the future, processing a lot of really long-term context window is really inefficient and also ineffective when you're trying to prompt these models for a really narrowly defined behavioral assets.”
Park: LLMs struggle with attention drop in the middle of long prompts
“Larger context window does confuse models, right? So we, some of my colleagues are actually doing more rigorous studies on this, where You can have a really long prompt, but model really focuses on the first few lines and the last few lines, and whatever comes…”