Sep 11, 2025 · 1h 3m · latent-space
Context Engineering for Agents - Lance Martin, LangChain
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space podcast, LangChain's Lance Martin explores the core principles of context engineering, detailing how developers can manage agent token flow, memory, retrieval, and multi-agent coordination while applying the Bitter Lesson as foundation models advance.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 16.3% of the talking time here. How this is scored →
The hosts as informed peer 6.1 Guest teaching 5.1 Guest disagreement 1.5 The hosts pushing back 1.7
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 37:35 Lance insists caching does not resolve context rot
Lance firmly pushes back against the premise that API caching solves long-context issues, distinguishing between latency/cost savings and fundamental model degradation over large token spans.
Hardest push from the hosts ▶ 36:31 Swyx challenges need for explicit message cachingSwyx immediately questions Lance's highlight of Manus caching, pointing out that modern LLM endpoints (OpenAI Responses API, Anthropic, Gemini) now perform implicit and automatic prefix caching.
Biggest teaching moment ▶ 12:35 Lance untangles read versus write multi-agent failure modesLance clearly educates the hosts on why multi-agent setups excel at read-heavy data collection (deep research) but degrade in write-heavy software development due to conflicting uncoordinated state decisions.
The host holds their own ▶ 33:52 Swyx demonstrates terminology authority via AI EngineerSwyx leverages his own creation and industry-wide adoption of the 'AI Engineer' taxonomy to validate and expand upon the guest's sociological theory of technical nomenclature.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Distinguishing Prompt Engineering from Context Engineering in Agent Workflows | 5 | 5 | 1 | 1 | Alessio asks Lance to differentiate prompt engineering from context engineering. Lance details the operational shift from single-turn chat prompts to managing multi-turn tool call trajectories in production agents. | |
| Context Offloading Strategies and Compression Trade-offs | 6 | 5 | 1 | 1 | Lance explains offloading raw tool outputs to external memory and using high-recall summarization to save tokens. Swyx connects this to findings from the Chroma Context Engineering meetup on compression and multi-agent roles. | |
| Context Isolation in Multi-Agent Systems: Read versus Write Paradigms | 5 | 7 | 2 | 1 | Alessio queries how context should be shared across sub-agents in coding workflows. Lance breaks down Cognition's critique of multi-agent systems, illustrating the structural difference between read-only parallel research and write-dependent coding. | |
| Retrieval Architecture Paradigms: Vector RAG versus Agentic File Search | 7 | 6 | 1 | 1 | The conversation contrasts classical vector RAG pipelines with agentic grep and file search. Swyx and Alessio contribute tools like DeepWiki and Context7 MCP while Lance walks through his benchmark showing llms.txt outperforming vector indices. | |
| Context Reduction Techniques, Pruning Risks, and Context Poisoning | 7 | 4 | 2 | 3 | Swyx brings up the active debate on whether an agent should prune failed trajectories or retain errors to learn from them. Lance responds by referencing Drew Breunig's work on context poisoning while weighing the maintenance overhead of custom pruning logic. | |
| Linguistic Evolution and the Sociological Emergence of AI Terminology | 6 | 3 | 1 | 1 | Lance cites Stewart Brand and Drew Breunig on why technical buzzwords take root anthropologically. Swyx validates the thesis by recounting his own rationale for coining the term AI Engineer. | |
| Message History Caching, Latency Optimization, and Context Rot Limits | 7 | 5 | 3 | 4 | Lance brings up Manus's message caching technique, but Swyx interjects that OpenAI, Anthropic, and Gemini already handle implicit prompt caching automatically. Lance clarifies that while caching solves latency and cost, it does not alleviate context rot. | |
| Agent Memory Architectures and Human-in-the-Loop Ambient Systems | 6 | 5 | 1 | 1 | Lance explains how memory decomposes into reading and writing automation, referencing Claude markdown files and Simon Willison's image generation bug. Lance emphasizes pairing memory with human-in-the-loop agent corrections. | |
| Applying The Bitter Lesson to Agent Architecture and Deep Research | 6 | 6 | 1 | 1 | Lance analyzes Hyung Won Chung's Stanford lecture on the Bitter Lesson, admitting he had to rewrite OpenDeepResearch twice to eliminate rigid inductive biases. Alessio and Swyx draw parallels to incumbents versus AI-native software architecture curves. | |
| Low-Level Orchestration Frameworks versus Restrictive Agent Abstractions | 6 | 5 | 2 | 3 | Swyx poses a direct question on whether LangGraph risks violating the Bitter Lesson. Lance defends low-level state-and-graph primitives (like Shopify's Roast) while agreeing that high-level black-box agent abstractions become obsolete liabilities. |
Statements from this episode (22)
Martin: Agent architectures are simple conceptually but managing context is hard
“When you kind of put together an agent, it's just tool clawing a loop. It's relatively simple to lay out, but it's actually quite tricky to get it to work well. In a particular, managing context with agents is a hard problem.”
Lance Martin: Prompt engineering is a subset of context engineering
“Well, I think that, you know, prompt engineering is kind of a subset of context engineering.”
Manus: A typical AI agent task requires approximately 50 tool calls
“Manus put out a great piece talking about content engineering with Manus, and they made the point that the typical Manus task is like 50 tool calls.”
Anthropic: Typical production agents execute hundreds of tool calls per task
“Anthropics multi-agent research is another nice example of this. They mentioned that the typical production agent, and this is probably referring to Cloud Code, could be other agents that they've produced, is like hundreds of tool calls.”
Martin: A naive deep researcher loop consumed 500,000 tokens per run
“My deep researcher, for example, with the naive tool calling loop was using 500,000 tokens. It was like a dollar to two dollars per run.”
Martin: Agents should offload raw tool context to external storage
“Rather than just writing back the full context of your tool calls, which could be token heavy, write those to disk and you can write back a summary. It could be a URL, something so that the agent knows it's retrieved a thing. It can fetch that on demand, but y…”
Martin: OpenDeep Research is the top-ranked open-source Deep Research agent
“OpenDeep Research is a deep research agent that I've been working on for about a year, and it's now, according to Deep Research Spence, the best performing Deep Research agent at least on that particular benchmark. So it's pretty good. Listen, it's not as good…”
Martin: Anthropic uses parallel sub-agents for research and single-shot final writing
“Anthropic reported on this too. So their deep researcher just uses parallelized subagents for research collation, and they do the writing in one shot at the end.”
Martin: Multi-agent systems excel at parallel read-only tasks, not writing tasks
“I like the take that apply multi-agents to problems that are easily parallelizable, that are read-only, for example, context gathering for deep research, and do, like, the final quote-unquote write, in this case report writing, at the end. I think this is tric…”
Martin: Claude Code operates entirely without codebase indexing
“Clock code doesn't do any indexing. It's just doing, quote unquote, agentic retrieval, just using simple tool calls, for example, using grep, to kind of poke around your files, no indexing whatsoever, and obviously works extremely well.”
Martin: Tool-based search with llms.txt beats maintaining vector indexes
“You give an LLM access to simple files, file tools. In this case, I actually use an LLM.txt to help it out. So it can actually know what's in each file. It's extremely effective and much more simple and easy to maintain, easier to maintain than building an ind…”
Martin: Claude Code triggers context compaction at 95% of context window
“If you use Cloud Code, you hit that
Kind of, you know, you've hit 95% of the context window, and you're about to, and Cloud Code's about to perform compaction.”
Martin: Selective History Pruning Adds Unnecessary Complexity to Agent Scaffolding
“So I'm not sure I love the idea of kind of selectively trying to prune your message history when you're building an agent. It can add more logic than you need to manage within your kind of agent scaffolding or harness.”
Martin: Context caching solves cost and latency, but not context rot
“I do think an important and subtle point here is that caching doesn't solve the long context problem. So it, of course, solves the problem of, like, latency and cost, but if you still have a 100,000 tokens in context whether it's cached or not, the LM is utili…”
Swix: Proprietary LLM caching creates vendor lock-in
“I feel like this is definitely a form of lock-in because you ideally want to be able to run prompts across multiple providers and all that. And yeah, caching is a hard problem. Like, I think ultimately, like, you control your destiny if you can run your own op…”
Martin: Cognition's Devin uses a fine-tuned model for context summarization
“Devin uses a fine-tuned model for doing summarization within the context of coding.”
Martin: Claude Code uses zero automation for reading and writing memory
“For reading memories, they just suck in your Claude MDs every time. So every time you spin up Claude Claude code, it pulls in all your Claude MDs. For writing memories, The user specifies, hey, I want to save this to memory, and then CloudCode writes it to Clo…”
Martin: Reading agent memory at scale is just a multi-step RAG pipeline
“Memory retrieval at large scale is just retrieval, right? I kind of view them as... In the case of sophisticated memory retrieval, it is just like a, you know, complex rag system in the same way we talked about with like Varun and building windsurf. It's kind …”
Martin: Agent memory pairs best with human-in-the-loop feedback
“Memory pairs really well with human in the loop. So for example, in my little email assistant, it's just an agent that runs my email, I have the opportunity to pause it before it sends off an email and correct it if I want, like, change the tone of this email,…”
Martin: AI engineers must continuously remove scaffolding as underlying models improve
“We should be adding structure necessary to get things to work today, but keeping an eye on improving models and keep, but keeping a close eye on models, improving rapidly and removing structure in order to un-bottleneck ourselves.”
Lance Martin: Black-box agent abstractions break when underlying models evolve
“Agent abstractions are problematic because you don't know what's necessarily under the hood of the abstraction. You don't understand it. And if I was building, for example, you know, OpenDeep research with an abstraction, I wouldn't necessarily know how to rip…”
Martin: The best open-source strategy pairs building tools with educational on-ramps
“I like this one, two punch of building a thing like open deep research, then creating a class so people can actually understand how to build it themselves. And I kind of like that build a thing, create an on-ramp for it.”