Liu: RAG will remain essential for managing LLM cost-performance trade-offs
Jerry Liu · RAG is a hack - with Jerry Liu of LlamaIndex · Oct 12, 2023 · at 15:01
Jerry Liu (creator of LlamaIndex) explains why larger LLM context windows will not eliminate the need for Retrieval-Augmented Generation (RAG).
0:00 / 0:14exact quote · 15.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“There's always going to be some curve regardless of like the performance of the best performing models of like cost versus performance. And so what RAG does is it does provide extra data points along that access because you kind of control the amount of context you actually wanted to retrieve.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jerry Liu
Liu: RAG is fundamentally just an algorithmic prompt-stuffing hack
“RAG is basically just a hack, but it turns out it's a very good hack because what is RAG? RAG is you keep the model fixed, and you just figure out a good way to, like, stuff stuff into the prompt of the language model. Everything that we're doing nowadays in t…”
Liu: Developers will eventually fine-tune new factual knowledge into LLMs
“That's one of those things where I think long-term, you definitely can. I think some people say you can't. I disagree. I think you definitely can. Just right now, I haven't gotten into work yet.”
Liu: Improving vector store lookup algorithms offers low marginal gains
“I don't think the delta on, like, improving the vector store, like, embedding lookup algorithm is that high. I think this stuff has been mostly solved or at least there's just a lot of other stuff you can do to try to improve the overall performance.”
Liu: The final state for personalized AI memory will not be RAG
“I think a lot of people have thoughts about that, but, like, for what it's worth, I don't think the final state will be ragged. I think it will be some, like, fancy algorithm or architecture where you, like, bake it into, like, the architecture of the model it…”