Context Rot

“I don't think it's like the galaxy brain software and we're very Far from there, because if you actually play with these tools a lot, context rod is real.”

“I do think an important and subtle point here is that caching doesn't solve the long context problem. So it, of course, solves the problem of, like, latency and cost, but if you still have a 100,000 tokens in context whether it's cached or not, the LM is utili…”

“If you just said, here's my entire code base, you know, go run wild, you know, to one agent, it's, it will just, you know, produce worse and worse code over time because it's going to have context rot.”

“And yeah, they're using context caching and that certainly helps, but like their cost and speed, but like isn't helping the context raw problem at all.”