context window

“Like, I think it's generally been shown that if you have the space to just put The raw files inside of a big context window. That is still better than chunking and retrieval. It just is.”

“In one sense, I think you're right that the throw everything into the context window thing is easier to maintain because you just can swap out a model. In another sense, it's, if things go wrong, it's harder to debug, where, like, if you know, here's the proce…”

“You could create, like, a long context dataset where, like, every single time the last 200 tokens can answer the entire question, and that's never gonna make the model attend to anything.”

“Making the model track state and have state management over time is really, really hard. And it's an incredibly hard evaluation that will probably only really work when you have a huge context.”

“Fundamentally to build a product like this, you need to get as much information in front of the model as possible and make sure that everything ever writes in output can be Traced back to something in the context window, so it's not hallucinating it.”

“Even getting to the, like, Ten million tokens of very fast and accurate context, which I expect to measure in, like, months, something like that.”

“We also have we have retrieval mechanisms, if required. So we natively try to use the context as much as it's available beyond which you know, we have, like, a rag setup to figure out”

“Just to add to that, I think like, just like a simple rule of thumb that we use is like, if it's the most recent set of research tasks where the user is likely to ask lots of follow-up questions, that should be in context. But like, as stuff gets 10 tasks ago,…”

“Like if you do have billion token context window model, you throw it all in there. It's still gonna be more expensive. The reason why retrieval is so important for us is because even if there is a model that's going to have these larger context windows, and ce…”

“To be honest, most people haven't complained about this. Like it comes up occasionally. But people tend to start new sessions pretty frequently. They tend to not have super long running things that often. So it is a problem that needs to be solved, but it's on…”

“The performance of LLMs is not invariant to how many tokens you use. As you use more and more tokens, the model can pay attention to less, and then also can reason sort of less effectively.”

“And so, like, I think pruning is also going to be, like, really, it's already becoming a thing, right? But, like, letting models, like, self-prune their context windows.”

“If you don't know what you're doing, and you don't really know what the AI model is capable of, and you don't have a lot of experience, like, you know, training wheels is like, when you get to 40%, start thinking about wrapping it up, or like doing a, like, yo…”

“But two is like, for the agentic tasks of 18 months from now, inside those major repositories of knowledge, and asking the models more and more things in underspecified ways, I suspect that the accuracy of the models would go down. The phenomenon of context fr…”