Prediction Open AI assessment confidence: 85% certainty 4/5 debate potential 3/5

Liu: Developers will eventually fine-tune new factual knowledge into LLMs

Jerry Liu · RAG is a hack - with Jerry Liu of LlamaIndex · Oct 12, 2023 · at 29:53

LlamaIndex founder Jerry Liu responds to the debate over whether LLMs can learn new factual knowledge via fine-tuning versus retrieval.

0:00 / 0:07exact quote · 7.6s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“That's one of those things where I think long-term, you definitely can. I think some people say you can't. I disagree. I think you definitely can. Just right now, I haven't gotten into work yet.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

In the long term, developers will definitely be able to fine-tune new factual knowledge into large language models, contrary to prevailing skepticism.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

Jerry Liu framed this prediction explicitly around the 'long term,' giving no specific deadline (the horizon date of 2028-10-12 is a pipeline default, not a speaker-set target). Under the ledger's default horizon rules, an open-ended prediction cannot be marked false merely because it has not yet materialized, unless it becomes provably impossible. As of September 2026, peer-reviewed research (e.g., EMNLP 2024 findings on fine-tuning inducing hallucinations, July 2026 work on the 'Knowing–Using Gap,' and NeurIPS 2025/2026 model editing benchmarks) and mainstream industry tooling still confirm that fine-tuning is ineffective and unreliable for injecting new factual knowledge relative to retrieval-augmented baselines. However, because the prediction remains an active area of ongoing long-term research, the claim is not yet determinable.

EMNLP 2024 study (Gekhman et al.) found fine-tuning LLMs on new factual knowledge slows acquisition and linearly increases hallucination rates.
July 2026 study (Dai et al.) identified a persistent 'Knowing–Using Gap,' where fine-tuned factual memorization fails to generalize into multi-hop reasoning circuits.
NeurIPS 2025 (UniEdit) and July 2026 (ForgetBench) benchmarks showed parametric model editing at scale causes catastrophic forgetting and collateral knowledge damage.
Industry consensus and practitioner guidance through 2026 continue to recommend RAG for factual updates while reserving fine-tuning for style, tone, and behavioral formatting.

Checked against

[1] aclanthology.org [2] arxiv.org [3] neurips.cc [4] emergentmind.com [5] alphaxiv.org [6] researchgate.net [7] arxiv.org [8] authorea.com

"fine-tuning" "factual knowledge" "catastrophic forgetting" "RAG" 2025 2026 fine tuning LLMs to learn new factual knowledge research benchmark "Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?" Gekhman ACL 2024 "model editing" "catastrophic forgetting" LLM benchmark 2024 2025 fine tuning vs RAG factual recall hallucination research 2024 2025 continual fine-tuning LLMs new factual knowledge reliable recall 2025 2026

assessed as of 2026-09-03

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

⏳ next review: 2028-10-12; checked again automatically on the next resolution wave

Mentioned

fine-tuning LLMs

⚑ Spot an error? Report it

More from Jerry Liu

Insight

Liu: RAG is fundamentally just an algorithmic prompt-stuffing hack

“RAG is basically just a hack, but it turns out it's a very good hack because what is RAG? RAG is you keep the model fixed, and you just figure out a good way to, like, stuff stuff into the prompt of the language model. Everything that we're doing nowadays in t…”

Jerry Liu Oct 12, 2023 ▶ 24:45 RAG is a hack - with Jerry Liu of LlamaIndex

Insight

Liu: RAG will remain essential for managing LLM cost-performance trade-offs

“There's always going to be some curve regardless of like the performance of the best performing models of like cost versus performance. And so what RAG does is it does provide extra data points along that access because you kind of control the amount of contex…”

Jerry Liu Oct 12, 2023 ▶ 15:01 RAG is a hack - with Jerry Liu of LlamaIndex

Insight

Liu: Improving vector store lookup algorithms offers low marginal gains

“I don't think the delta on, like, improving the vector store, like, embedding lookup algorithm is that high. I think this stuff has been mostly solved or at least there's just a lot of other stuff you can do to try to improve the overall performance.”

Jerry Liu Oct 12, 2023 ▶ 1:01:34 RAG is a hack - with Jerry Liu of LlamaIndex

Prediction Not checkable as stated

Liu: The final state for personalized AI memory will not be RAG

“I think a lot of people have thoughts about that, but, like, for what it's worth, I don't think the final state will be ragged. I think it will be some, like, fancy algorithm or architecture where you, like, bake it into, like, the architecture of the model it…”

Jerry Liu Oct 12, 2023 ▶ 1:10:29 RAG is a hack - with Jerry Liu of LlamaIndex

Insight

Liu: Avoid pre-GPT-4 models for tasks requiring complex reasoning

“Like, I'm one of the first to say, like, you know, you shouldn't use anything pre-GPT-IV for anything that requires, like, complex reasoning because it's just going to gonna be unreliable. Okay, disregarding stuff like fine-tuning.”

Jerry Liu Oct 12, 2023 ▶ 9:28 RAG is a hack - with Jerry Liu of LlamaIndex

Opinion

Liu: Security and access control are not P0 for enterprise apps

“I think users have asked for it, but I don't think that's like a P zero. Like, I think the P zero is more on, like, can we get this thing working before we expand this to, like, more users within the org.”

Jerry Liu Oct 12, 2023 ▶ 34:32 RAG is a hack - with Jerry Liu of LlamaIndex

Liu: Developers will eventually fine-tune new factual knowledge into LLMs

More from Jerry Liu

Turn any episode into a week of clips.