Huber: LLMs will largely replace purpose-built re-rankers
Jeff Huber · Long Live Context Engineering - with Jeff Huber of Chroma · Aug 19, 2025 · at 25:50
Jeff Huber, co-founder and CEO of Chroma, explains why brute-forcing retrieval re-ranking via LLMs will supersede specialized re-ranking models.
0:00 / 0:38exact quote · 38.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think that, like, this is going to be the dominant paradigm. I actually think that, like, probably purpose-built re-rankers will go away, and the same way that, like, purpose-built, they'll still exist, right? Like, if you're at extreme scale, extreme cost, yes, you'll care to optimize that. And the same way that if you're running with hardware, right? Like, you're just going to use a CPU or a GPU, unless you absolutely have to have an ASIC or an FPGA. And I think the same thing is true about, like, re-rankers, where, like, As LLMs become a hundred, a thousand times faster, a hundred, a thousand times cheaper, that like people are just going to use LLMs for re-rankers and that actually like brute forcing information, curation is going to become extremely, extremely popular.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jeff Huber
Huber: Silicon Valley treats AGI as a secular religion
“I think AGI is also a religion. It has a problem of evil. We don't have enough intelligence. It has a solution, a deus ex machina. It has the second coming of Christ that AGI, the singularity is going to come. It's going to save humanity because we will now ha…”
Huber: Frontier models repeat mistakes if failed actions remain in context
“A few of the insights is, like, everyone, frontier model is not good at search. Humans have this natural explore-exploit trade-off, where we kind of understand, like, when to stop doing something. Also, humans are pretty good at, like, forgetting, actually, li…”
Huber: LLM Performance and Reasoning Degrade as Token Counts Increase
“The performance of LLMs is not invariant to how many tokens you use. As you use more and more tokens, the model can pay attention to less, and then also can reason sort of less effectively.”
Huber: Regex handles 90% of code queries; embeddings add marginal improvement
“My guess is that, like, for code today, it's something like, 90% of queries or 85% of queries can be satisfactorily run with regex. Regex is obviously, like, the dominant pattern used by Google code search, GitHub code search, but you maybe can get, like, 15% …”
Huber: Future retrieval systems will operate entirely within latent space
“I think, like, there's a few things that I think might be true about retrieval systems in the future. So, like, number one, they just stay in latent space, they don't go back to natural language.”
Huber: RBAC is dead and AI vendors are ignoring authorization
“Well, I'm just saying, I think, like, RBAC does seem to be dead, right? Like, if you want to say something is dead, probably RBAC is dead. And like the auth story to me seems like incredibly unsolved and unaddressed by like the existing state of like AI vendor…”