Huber: LLM Performance and Reasoning Degrade as Token Counts Increase
Jeff Huber · Long Live Context Engineering - with Jeff Huber of Chroma · Aug 19, 2025 · at 14:05
Jeff Huber is the co-founder and CEO of Chroma. He is discussing Chroma's research report on context rot and why managing token context is critical for AI systems.
0:00 / 0:09exact quote · 9.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The performance of LLMs is not invariant to how many tokens you use. As you use more and more tokens, the model can pay attention to less, and then also can reason sort of less effectively.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jeff Huber
Huber: Silicon Valley treats AGI as a secular religion
“I think AGI is also a religion. It has a problem of evil. We don't have enough intelligence. It has a solution, a deus ex machina. It has the second coming of Christ that AGI, the singularity is going to come. It's going to save humanity because we will now ha…”
Huber: Frontier models repeat mistakes if failed actions remain in context
“A few of the insights is, like, everyone, frontier model is not good at search. Humans have this natural explore-exploit trade-off, where we kind of understand, like, when to stop doing something. Also, humans are pretty good at, like, forgetting, actually, li…”
Huber: LLMs will largely replace purpose-built re-rankers
“I think that, like, this is going to be the dominant paradigm. I actually think that, like, probably purpose-built re-rankers will go away, and the same way that, like, purpose-built, they'll still exist, right? Like, if you're at extreme scale, extreme cost, …”
Huber: Regex handles 90% of code queries; embeddings add marginal improvement
“My guess is that, like, for code today, it's something like, 90% of queries or 85% of queries can be satisfactorily run with regex. Regex is obviously, like, the dominant pattern used by Google code search, GitHub code search, but you maybe can get, like, 15% …”
Huber: Future retrieval systems will operate entirely within latent space
“I think, like, there's a few things that I think might be true about retrieval systems in the future. So, like, number one, they just stay in latent space, they don't go back to natural language.”
Huber: RBAC is dead and AI vendors are ignoring authorization
“Well, I'm just saying, I think, like, RBAC does seem to be dead, right? Like, if you want to say something is dead, probably RBAC is dead. And like the auth story to me seems like incredibly unsolved and unaddressed by like the existing state of like AI vendor…”