Context Window

“But two is like, for the agentic tasks of 18 months from now, inside those major repositories of knowledge, and asking the models more and more things in underspecified ways, I suspect that the accuracy of the models would go down. The phenomenon of context fr…”

“We're not seeing that with AI models today. They kind of, they're born into a world for, and they exist for a very short context window, and then they just, like, disappear. And yeah, there are things that you can kind of do to, like, continue them, but it's v…”

“Even though we're working on machines, not biological brains, and so potentially you could have, you know, millions or tens of millions size context window or memory, and it can be perfect. There's still a cost to looking it up and finding the right thing that…”

“And then the problem is if you're then trying to try and process live video, and you're just going to naively record all the tokens, then actually a million tokens isn't that much. It's only like 20 minutes.”

“Cause the model will get dumb as the context window closes, right? So if you have like a context window and I can draw this out, if this is your context window and like the optimal is You're between, like, there's always, like, maybe, like, 10% is already fill…”

“And so then thus, if anything can ever enter that context window, because they have access to a resource, then in theory you should assume it can be, you know, prompt ejected out of the context window, and I don't know that we know of a way to solve that at th…”

“And the way that I think about a file system is it basically lets the LLM manage its own context window. So it can decide what to read from files.”

“If you don't know what you're doing, and you don't really know what the AI model is capable of, and you don't have a lot of experience, like, you know, training wheels is like, when you get to 40%, start thinking about wrapping it up, or like doing a, like, yo…”

“And so, like, I think pruning is also going to be, like, really, it's already becoming a thing, right? But, like, letting models, like, self-prune their context windows.”

“And that I think that was the big change in and realization through vibe coding is that agents are good enough at manipulating file systems that they can use this as a replacement for their Context window, basically. They can select parts of what they want to …”

“If there's a good quote I've read, I can't, I apologize to the author cause I can't attribute it off the top of my head, but it's like the ceiling on the AI isn't the model intelligence. It's what the model sees before it acts. So that's the ceiling right now.…”

“I mean, I still think context window is like probably the number one limit. Like if you look at cloud code executing, it's delegating to all these different context windows. At the end of the day, when each one comes back, it's like getting some sort of summar…”

“I think we're experiencing memory through putting really good summaries into the context window, right, or into the prompt.”

“I would say LLMs today, you know, they're marketed as a million token length. Which is like a million words almost. In reality, it's about 200,000, and then they start to struggle.”

“The performance of LLMs is not invariant to how many tokens you use. As you use more and more tokens, the model can pay attention to less, and then also can reason sort of less effectively.”

“To be honest, most people haven't complained about this. Like it comes up occasionally. But people tend to start new sessions pretty frequently. They tend to not have super long running things that often. So it is a problem that needs to be solved, but it's on…”

“LLMs don't natively do this, and this is not something that has really been solved in the R&D of LLMs, I think. And so context windows are really kind of like working memory, and you have to sort of program the working memory quite directly because they don't …”

“The one thing that you can rely on in terms of the direction of AI progress is the context windows will expand, right? You'll be able to feed more into the prompt and get a useful output right now.”

“Like if you do have billion token context window model, you throw it all in there. It's still gonna be more expensive. The reason why retrieval is so important for us is because even if there is a model that's going to have these larger context windows, and ce…”

“And I think the number one problem in all of AI is elongating the context window.”

“I mean, I think there's always this thought of that, like, you know, the models will have near infinite context windows and you can just give them everything and they can figure things out automatically. But I don't think, you know, like they're anywhere close…”

“We also have we have retrieval mechanisms, if required. So we natively try to use the context as much as it's available beyond which you know, we have, like, a rag setup to figure out”

“Just to add to that, I think like, just like a simple rule of thumb that we use is like, if it's the most recent set of research tasks where the user is likely to ask lots of follow-up questions, that should be in context. But like, as stuff gets 10 tasks ago,…”

“You wouldn't want An AGI to jam a 100,000 documents into its context window and take an infinite amount of time to, or some very, very large amount of time to process that. You'd rather use and orchestrate a bunch of sub-agents.”

“Google Gemini said, oh, we have a two million token context window, and that sounded amazing until you realize, like, that's one five megabyte file.”

“Even getting to the, like, Ten million tokens of very fast and accurate context, which I expect to measure in, like, months, something like that.”

“Technically a page is also a component, but the smaller you go, the more you can iterate and focus on that without overwhelming the context window. And that's why I think it's like a lot more effective.”

“Fundamentally to build a product like this, you need to get as much information in front of the model as possible and make sure that everything ever writes in output can be Traced back to something in the context window, so it's not hallucinating it.”

“You could create, like, a long context dataset where, like, every single time the last 200 tokens can answer the entire question, and that's never gonna make the model attend to anything.”

“Making the model track state and have state management over time is really, really hard. And it's an incredibly hard evaluation that will probably only really work when you have a huge context.”

“Well, our AI systems today are kind of operating with short term memories that are millions of numbers in length, and it feels very unintuitive. Ultimately, we want them to instead be able to bake stuff into some kind of long term storage, and that's going to …”

“And then it seems lucky that a lot of people will end up in the ten million plus range in the next a year or two or, you know, some reasonable timeframe ahead.”

“In one sense, I think you're right that the throw everything into the context window thing is easier to maintain because you just can swap out a model. In another sense, it's, if things go wrong, it's harder to debug, where, like, if you know, here's the proce…”

“There are also things where it requires iterations. You need to like decide what to do, interact with the environment, get that back. So this whole idea of chaining and agents, I don't like, That's less around context windows and more around interacting with t…”

“I'm more of the belief that it just opens up the set of trade-offs you can make between retrieval, more sophisticated retrieval and model reasoning by having a larger context window versus saying, like, we don't need any ability to work with a specific data se…”

“The context window is not correlated to the quality of the model outputs.”

“Like, I think it's generally been shown that if you have the space to just put The raw files inside of a big context window. That is still better than chunking and retrieval. It just is.”

“Traditionally, you'd say this is very, very inefficient, and it actually means that the larger your context window, aka token limit, aka prompt length, gets, the more computationally expensive it gets on a quadratic basis. So doubling your input means quadrupl…”

“The idea that like a hundred K or even a million tokens is going to solve all our problems. And like, you just ask your questions and dump in all context naively is ridiculous to me. I think it's a really important area of product and research work.”

“I think in two years it'll be reasonably solved in terms of the size of context windows. It'll be usable. Like, I'd be happy to take that bet, but we'll see. Like, I think the context windows are going to get very large.”