Jun 6, 2024 · 36m · no-priors
No Priors Ep. 67 | With Voyage AI Co-Founder and CEO
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of No Priors, Sarah Guo interviews Stanford professor and Voyage AI CEO Tengyu Ma about the economics, technical design, and algorithmic optimization of Retrieval-Augmented Generation (RAG) systems. Ma breaks down the RAG versus long-context debate, introduces training efficiency breakthroughs like the Sophia optimizer, and shares insights on bridging academic research with AI entrepreneurship.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 23% of the talking time here. How this is scored →
The hosts as informed peer 5.9 Guest teaching 3.8 Guest disagreement 1.2 The hosts pushing back 1.3
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 18:09 Reframing agent chaining as dependent on retrieval
Tengyu rejects the premise that agent chaining displaces retrieval systems, arguing instead that multi-step agents fundamentally rely on embeddings and small models to remain computationally viable.
Hardest push from the hosts ▶ 10:42 Challenging RAG with infinite context and agent architecturesSarah directly frames the leading counterarguments from top frontier labs questioning whether RAG architectures will be rendered obsolete by infinite context windows and agentic chaining.
Biggest teaching moment ▶ 13:00 First-principles memory hierarchy comparisonTengyu educates listeners on the theoretical cost and memory bottlenecks of full-context transformers, using computer architecture caching levels to prove why retrieval remains essential.
The host holds their own ▶ 16:12 Translating token metrics to concrete engineering limitsSarah demonstrates deep technical and operational domain knowledge by calculating the real-world scale limits of 1M token windows against enterprise codebases and media requirements.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Tengyu Ma's Research Agenda and Focus Areas | 6 | 4 | 1 | 1 | Sarah shows strong familiarity with Tengyu's background across theory, RL, and optimizers, interjecting with industry context about Adam's age. Tengyu explains his academic evolution from matrix completion to contrastive learning and the Sophia optimizer. | |
| Founding Voyage AI and the Evolution of Applied ML | 5 | 3 | 1 | 1 | Sarah connects Tengyu's transition into entrepreneurship to Conviction's investment thesis about foundation models commoditizing earlier pipeline steps. Tengyu describes the historical shift from 7-step applied ML workflows to prompt and RAG architectures. | |
| Understanding RAG Systems and Core Retrieval Components | 5 | 4 | 1 | 1 | Sarah prompts an architectural breakdown of RAG systems and chimes in on vectorizing diverse modalities like code. Tengyu outlines the core mechanics of embedding vectorization, retrieval grounding, and hallucination reduction. | |
| Real-World Applications and Enterprise Use Cases for RAG | 7 | 5 | 2 | 3 | Sarah poses the industry counterarguments against RAG, citing agent chaining and infinite-context LLMs. Tengyu addresses the debate systematically, using hardware caching analogies to argue that hierarchical retrieval remains vastly more cost-efficient than long context. | |
| Token Limits and Scale in Enterprise Contexts | 7 | 3 | 1 | 1 | Sarah contextualizes Gemini 1.5 Pro's 1M token window in practical terms (code lines, book lengths) and articulates why enterprise scale still demands retrieval. Tengyu reinforces the math with enterprise cost multiples. | |
| Agent Chaining and Iterative Retrieval Approaches | 6 | 4 | 2 | 2 | Sarah probes agent chaining as an alternative data management approach. Tengyu reframes agent chaining as orthogonal, explaining that agents still require embedding models and iterative retrieval for efficiency. | |
| Optimizing Retrieval Performance and Pipeline Simplicity | 5 | 4 | 1 | 1 | Sarah asks how builders should optimize RAG pipelines beyond the core LLM. Tengyu explains his vision where smarter foundation models eliminate brittle software heuristics like document chunking. | |
| Domain-Specific Fine-Tuning and Latency Budgets | 7 | 4 | 1 | 1 | Sarah explains inference-time latency mechanics in search pipelines to clarify why parameter count matters. Tengyu provides benchmarks on domain-specific fine-tuning across code and legal corpora. | |
| Customization for Enterprises and Practical Builder Advice | 6 | 3 | 1 | 1 | Sarah prompts practical recommendations for developers and asks for forward-looking predictions on model evolution. Tengyu details profiling strategies and predicts simplified 3-4 component architectures. | |
| Lessons Learned Transitioning from Academia to Founder | 5 | 4 | 1 | 1 | Sarah questions the role of academic labs amid massive industrial scaling laws. Tengyu outlines why universities must target 3-to-5-year breakthrough horizons like fundamental optimizers and reasoning conjectures rather than short-term scale. |
Statements from this episode (19)
Ma: AI Is Running Out of Compute and Training Data
“My vision is that in the future the efficiency is very important because we are running out of data and compute. So we have to either use the data much better and use the compute much better.”
Ma: Sophia Optimizer Improves LLM Pre-Training Efficiency by 2x
“One of the paper we wrote last year was Sophia which we found, where we found that we have a nutrient optimizer, which can improve the training efficiency by two X for pre-training.”
Ma: Meta Achieved 1.6x Training Efficiency Gain Using Sophia Optimizer
“And recently I think one of the Facebook friends actually used this in their large scale multimodal training. And they found that in on that scale, I don't know exactly how many parameters there are, but I think, I assume it's kind of more than a hundred billi…”
Ma: RAG Response Quality Is Bottlenecked by Retrieval Quality
“Right now for implementing Rack the bottleneck seems to be that, you know, it's not very hard to implement it, right? You can just connect the components and have your Rack system ready very quickly. But the bottleneck seems to be the quality. Of the response …”
Ma: Fine-Tuning Often Fails Due to Data Demands and Hallucinations
“Fine tuning in many cases doesn't work because you need a lot of data to see the results and there are still hallucinations even after fine tuning.”
Ma: RAG Will Remain Much Cheaper Than Long-Context Windows
“My prediction is that reg will be much cheaper than long contacts going forward.”
Ma: 100M-Token Enterprises Cannot Afford Long-Context Inference Costs
“So right now, if they have a hundred million tokens, I don't think they can use long context transformers at all because it's way too expensive.”
Ma: Agent Chaining Architectures Still Rely on Embedding Models
“On the first level bit I would say is that I think it's kind of orthogonal to embedding models and re-rankers to some degree, because even when you have agent chaining, right, you still probably use embedding models as part of the chain, right?”
Ma: Iterative Retrieval Will Diminish as Embedding Models Improve
“However, in the long run, my suspicion is that iterative retrieval will be useful, but it will be a bit less useful as the If the embedding models becomes more and more clever, right? So once the embedding models are more clever, then maybe one run or two runs…”
Ma: Voyage AI Trains Embedding Models on Trillions of Tokens
“We train our network on trillions of tokens at least and we fine tune them for special use cases.”
Ma: RAG Software Heuristics Will Vanish as Embedding Models Improve
“And my long term vision here is that some of the software engineering layers on top of the networks will be less and less needed when the networks are more and more clever.”
Ma: Multimodal Embeddings Will Eliminate Image-to-Text Preprocessing
“So maybe in the future, you don't have to turn your images into description of images and then give it to the text embedding model. That's what people are doing right now. Like everything is turned into text and then use a text embedding model. But when the em…”
Ma: Voyage Fine-Tuned Models on 2T Code and 1T Legal Tokens
“We fine tune on two trillions of code snippets, tokens, and then we get a code embedding model and we do the fine tuning on one trillion legal tokens.”
Ma: Latency Limits Embedding Models to 10 Billion Parameters
“Basically it's impossible to use more than ten billion parameters. For embedding models.”
Ma: Domain-Specific Fine-Tuning Yields 15-20% Retrieval Gains in Code
“And we have seen like five to 20% of improvements by this domain specific Fine-tuning depending on the particular domains. For code, we have seen 15 to 20% of improvement, partly because we have a lot of data there.”
Ma: Voyage AI Produces Embeddings 3-4x Smaller Than Competitors
“So we produce embeddings that is like a three X, you know, four X smaller dimension than some of the competitors.”
Ma: Proprietary Data Fine-Tuning Adds 10-20% Retrieval Accuracy
“So we fine tune on the proprietary data of a particular company, and we can see 10 to 20% improvement on top of the domain specific in fine tuning as well.”
Ma: Academic AI Labs Should Avoid Competing on Model Scaling
“My view is that I think academia probably should work on some different questions from what industry is good at, right? So if we are just only working on how to scale up the system, then obviously the incentive is not right. The, we don't have enough capital t…”
Ma: Scaling Web Data Unlikely to Solve Complex Math Conjectures
“It's very unclear whether you can really the scaling law is really enough to get Get you to prove Riemann hypothesis or any of the math conjectures. So you know, and also you have to be superhuman performance in some sense, right? So if you turn on just the co…”