Ma: RAG Will Remain Much Cheaper Than Long-Context Windows
Tengyu Ma · No Priors Ep. 67 | With Voyage AI Co-Founder and CEO · Jun 6, 2024 · at 14:11
Tengyu Ma, CEO of Voyage AI, compares the economics of RAG against feeding full proprietary datasets into long-context LLM windows.
0:00 / 0:04exact quote · 4.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“My prediction is that reg will be much cheaper than long contacts going forward.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Tengyu Ma
Ma: AI Is Running Out of Compute and Training Data
“My vision is that in the future the efficiency is very important because we are running out of data and compute. So we have to either use the data much better and use the compute much better.”
Ma: Sophia Optimizer Improves LLM Pre-Training Efficiency by 2x
“One of the paper we wrote last year was Sophia which we found, where we found that we have a nutrient optimizer, which can improve the training efficiency by two X for pre-training.”
Ma: Meta Achieved 1.6x Training Efficiency Gain Using Sophia Optimizer
“And recently I think one of the Facebook friends actually used this in their large scale multimodal training. And they found that in on that scale, I don't know exactly how many parameters there are, but I think, I assume it's kind of more than a hundred billi…”
Ma: Fine-Tuning Often Fails Due to Data Demands and Hallucinations
“Fine tuning in many cases doesn't work because you need a lot of data to see the results and there are still hallucinations even after fine tuning.”
Ma: 100M-Token Enterprises Cannot Afford Long-Context Inference Costs
“So right now, if they have a hundred million tokens, I don't think they can use long context transformers at all because it's way too expensive.”
Ma: Iterative Retrieval Will Diminish as Embedding Models Improve
“However, in the long run, my suspicion is that iterative retrieval will be useful, but it will be a bit less useful as the If the embedding models becomes more and more clever, right? So once the embedding models are more clever, then maybe one run or two runs…”