Martin: Tool-based search with llms.txt beats maintaining vector indexes
Lance Martin · Context Engineering for Agents - Lance Martin, LangChain · Sep 11, 2025 · at 20:54
Lance Martin of LangChain discusses his benchmark results comparing vector store retrieval against lightweight agentic file search using llms.txt files.
0:00 / 0:13exact quote · 13.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“You give an LLM access to simple files, file tools. In this case, I actually use an LLM.txt to help it out. So it can actually know what's in each file. It's extremely effective and much more simple and easy to maintain, easier to maintain than building an index.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Lance Martin
Martin: AI engineers must continuously remove scaffolding as underlying models improve
“We should be adding structure necessary to get things to work today, but keeping an eye on improving models and keep, but keeping a close eye on models, improving rapidly and removing structure in order to un-bottleneck ourselves.”
Martin: OpenDeep Research is the top-ranked open-source Deep Research agent
“OpenDeep Research is a deep research agent that I've been working on for about a year, and it's now, according to Deep Research Spence, the best performing Deep Research agent at least on that particular benchmark. So it's pretty good. Listen, it's not as good…”
Martin: Multi-agent systems excel at parallel read-only tasks, not writing tasks
“I like the take that apply multi-agents to problems that are easily parallelizable, that are read-only, for example, context gathering for deep research, and do, like, the final quote-unquote write, in this case report writing, at the end. I think this is tric…”
Martin: Selective History Pruning Adds Unnecessary Complexity to Agent Scaffolding
“So I'm not sure I love the idea of kind of selectively trying to prune your message history when you're building an agent. It can add more logic than you need to manage within your kind of agent scaffolding or harness.”
Martin: Context caching solves cost and latency, but not context rot
“I do think an important and subtle point here is that caching doesn't solve the long context problem. So it, of course, solves the problem of, like, latency and cost, but if you still have a 100,000 tokens in context whether it's cached or not, the LM is utili…”
Lance Martin: Black-box agent abstractions break when underlying models evolve
“Agent abstractions are problematic because you don't know what's necessarily under the hood of the abstraction. You don't understand it. And if I was building, for example, you know, OpenDeep research with an abstraction, I wouldn't necessarily know how to rip…”