Huber: Language models require a multi-tiered memory hierarchy like traditional computers
Jeff Huber · The Most UNDERRATED Use Case in AI | Jeff Huber on TBPN April 8th · Apr 10, 2025 · at 1:13
Jeff Huber, co-founder of Chroma, discusses the architectural debate between expanding LLM context windows and using retrieval-augmented generation (RAG).
0:00 / 0:25exact quote · 25.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“In the same way that we have a memory hierarchy in classic computers, right, we have the CPU, RAM, disk, and network we are also going to have a similar memory hierarchy in language models. And again, it already exists today. We have the actual sort of transformer attention heads. We have the context window. We have the retrieval system and tool use, and these things have different trade-offs, right? You think about kind of access, speed, capacity, and cost.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jeff Huber
Huber: LLMs are like CPUs, not operating systems
“I don't think of an LLM as an operating system. I think an LLM is much more like a CPU, right? It's an information processing unit.”
Huber: Open-source models win B2B through developer focus, not beating GPT-5
“Focus on the developers. I think that's the beachhead. That's how you win the B to B market. If you win the B to B market with your open source models, Like, you get all of the sort of downstream effects that you want. You know, you don't need to beat you know…”
Huber: 10x compute increases are not producing 10x better AI models
“Diminishing, they're clearly diminishing marginal returns, right? We're sort of spending 10 X on compute. We're not getting 10 X or better models, at least evidently not yet.”
Huber: AI will probably drive GDP growth exceeding the Industrial Revolution
“It's a, you know, technology is probably as important as the invention of electricity. It will probably, you know, bring about a increase in GDP that is on the order of the industrial revolution or greater.”
Huber: In 10 years, the poorest could have better healthcare than today's billionaires
“Like it is very possible the poorest people on earth today, or, you know, in 10 years, we'll have access to better healthcare better legal representation you know, better financial services than, like, billionaires have today.”