Sep 27, 2024 · 1h 26m · latent-space
Language Agents: From Reasoning to Acting — with Shunyu Yao of OpenAI, Harrison Chase of LangGraph
Shunyu Yao · 45m spoken Harrison Chase · 18m spoken Shawn Wang · 10m spoken Alessio Fanelli · 6m spoken
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
LangChain creator Harrison Chase and researcher Shunyu Yao join the Latent Space podcast to discuss the evolution, cognitive architectures, evaluation benchmarks, and tooling design powering modern language agents. They explore foundational concepts from ReAct and Reflexion to Agent-Computer Interfaces (ACI) and stateful production orchestration with LangGraph.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 20.7% of the talking time here. How this is scored →
The hosts as informed peer 6.5 Guest teaching 5.2 Guest disagreement 1.7 The hosts pushing back 2.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 57:19 Shunyu rejecting complete intelligence-knowledge separation
Shunyu directly challenges Swyx's thesis on isolating intelligence from knowledge, arguing that intelligence inherently emerges through acquired knowledge rather than existing in isolation.
Hardest push from the hosts ▶ 47:21 Swyx pushing back on HCI and ACI equivalenceSwyx challenges the idea that interfaces designed for humans cleanly translate to AI agents, providing detailed counterexamples from structured output schemas and verbose compiler error loops.
Biggest teaching moment ▶ 51:42 Shunyu breaking down fundamental memory limits of ACI vs HCIShunyu educates the hosts on the foundational cognitive contrast between narrow human working memory (requiring sequential steps) and expansive LLM context windows (benefiting from parallel batch results).
The host holds their own ▶ 47:31 Swyx detailing specific schema field optimizations for LLM reasoningSwyx demonstrates hands-on engineering expertise by showing how altering JSON key names to candidate topics triggers better chain-of-thought behavior in frontier models.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Welcome and Introductions with Harrison Chase and Shunyu Yao | 6 | 5 | 1 | 1 | The hosts open with deep familiarity regarding Shunyu Yao's PhD defense and Harrison Chase's early inspiration from the ReAct paper. Shunyu provides historical context on text-adventure games like Zork I and the shift from RL to LLM-driven reasoning. The exchange is warm, collaborative, and appreciative. | |
| ReAct's Legacy, Tool Calling, and the Modern Agent Loop | 6 | 5 | 2 | 2 | Harrison and Shunyu discuss whether modern function-calling loops still reflect ReAct principles. Shunyu clarifies that inner monologues remain crucial when tool APIs diverge from pre-training data distributions, while Alessio asks how thinking steps are being internalized into base model weights. | |
| Reflexion, Language-Based Feedback, and Agent Memory Architectures | 7 | 6 | 1 | 2 | Shunyu explains Reflexion as substituting scalar RL rewards with rich linguistic feedback acting as linguistic gradient descent. The hosts and Harrison connect this to LangMem, Voyager, and cognitive science categorizations of semantic versus procedural memory. | |
| Tree of Thoughts, Search Algorithms, and Prompting Philosophy | 6 | 4 | 2 | 2 | The discussion covers Tree of Thoughts, search vs. reactive tasks, and the minimalist prompting philosophy. Shunyu rejects complex emotional prompt hacks in favor of plain human-to-human communication, prompting Harrison and Swyx to analyze user communication hurdles. | |
| MCTS, The Benchmark Bottleneck, and Evaluation Realism | 6 | 6 | 2 | 1 | Shunyu critiques the academic habit of applying over-complex methods to trivial benchmarks rather than building scalable, realistic tasks. He unpacks the trilemma of benchmark design between auto-gradability, realism, and scalability. | |
| Interactive Coding, SWE-Agent, and Agent-Computer Interfaces (ACI) | 7 | 6 | 1 | 2 | Shunyu details the progression from InterCode to SWE-bench and SWE-agent, highlighting that optimizing the Agent-Computer Interface (ACI) yields far higher returns than raw planning algorithms. The hosts draw parallels to Cognition's Devin and user interaction paradigms. | |
| Optimizing Interfaces for LLMs: ACI vs. HCI and Machine Limits | 8 | 5 | 3 | 4 | Swyx pushes back against the premise that HCI and ACI design completely overlap, citing his empirical experience with structured output field naming and verbose error traces. Shunyu contrasts human working memory constraints with machine context limits, while Harrison argues for allowing agent interfaces to diverge early. | |
| Separating Intelligence from Knowledge and Training Trajectory Data | 7 | 6 | 3 | 3 | Swyx poses whether intelligence can be cleanly separated from knowledge, pointing to on-device LoRA adapters. Shunyu pushes back, arguing from historical AI and Hinton's perspective that intelligence emerges with knowledge, though framing knowledge as a cache for intelligence. | |
| CoALA Framework, Action Spaces, and Stateful Orchestration | 7 | 5 | 1 | 2 | Shunyu walks through the CoALA framework across memory, action space, and decision-making. Harrison and Swyx integrate LangGraph's stateful cross-thread persistence model into the architecture, debating whether developers or agents should select memory tooling. | |
| Tau-Bench, Customer Simulation, and Practical Enterprise Agents | 6 | 5 | 1 | 1 | Alessio and Shunyu discuss Tau-Bench and customer service agent simulation where LLMs model realistic user behavior under information asymmetry. Harrison answers Shunyu's question about enterprise applications, pointing to customer support, SDR data enrichment, and spreadsheet agents. | |
| LangGraph Studio, Agent IDEs, and Future Tooling | 6 | 4 | 2 | 2 | Shunyu questions whether low-code tooling is truly ready for non-programmers, and Harrison clarifies that LangGraph Studio acts as an IDE to bridge developer architecture with PM prompt refinement. The group wraps up with reflections on future agent UX. |
Statements from this episode (30)
Shunyu Yao says Ilya Sutskever claimed GPT-1 had solved language
“Back in OpenAI, they did this GPT-ONE together, and Ilya just said, Karthik, you should stay, because we just solved the language.”
Shunyu Yao built the ReAct prototype before Chain-of-Thought existed
“The prototype I think was around November of 2021. So that's even before like chain of thought or whatever came up.”
Yao: Text adventure games remain very hard even for GPT-4
“Like those texting are just too hard. I think today it's still very hard. Like if you used to be before to solve it, it's still very hard.”
Harrison Chase says production AI agents rely on three main defaults
“And there's such a long tail of other ones, but in practice, like, when people go to production, they generally have their own tools, or maybe one of those three, maybe some other ones, but, like, very, very few other ones.”
Yao: Pairing reasoning with tool use is essential for unfamiliar tools
“And I think the second contribution is this idea of what people call like inner monologue or thinking or reasoning or whatever to be paired with tool use. I think that's still not trivial because if you look at the default function calling or whatever, like th…”
Harrison Chase says OpenAI recommends adding a thought field to tool schemas
“I think open AI even recommended, like when you're doing tool calling, it's sometimes helpful to put like a thought field in the tool along with all the actual acquired arguments and then have that one first. So it fills out that first and then, and that's, th…”
Yao: Reflexion replaces scalar RL rewards with verbal gradient descent
“I think one way to think of reflection is that the traditional idea of reinforcement learning is you have a scalar reward, and then you somehow back propagate the signal of the scalar reward. To the rest of your neural network through whatever algorithm, like …”
Chase: LangChain and other frameworks lack off-the-shelf Reflexion implementations
“I don't think we have like an off the shelf kind of like implementation of reflection and kind of like the general sense. I think the concepts like absolutely we see used in different kind of like specific cognitive architectures, but I don't think we have one…”
Yao: Evaluator quality is the key bottleneck for agent self-reflection
“I think a key bottleneck is the evaluator, right? Basically you need to have a good sense of the signal. So for example, like if you are trying to do a very hard reasoning task, say mathematics, For example, and you don't have any tools, right? It's operating …”
Harrison Chase says ReAct is the most popular agent prompting framework
“I would say like reacts probably like the most popular. I think there's aspects of reflection that Get used. Tree of thought, probably like the least so.”
Shunyu Yao advises developers to default to minimalist prompting for AI agents
“And I think in terms of the actual prompting method to use for a particular problem, I'm I think we should all be in the minimum list kind of camp, right? You should try the minimum thing and see if it works and if it doesn't work and there's absolute reason t…”
Shunyu Yao argues modern LLMs make prompt engineering tricks obsolete
“I feel like in some sense, I feel like prompt engineering, even it's like a slightly negative word at the time, because it refers to all those kind of weird tricks that you have to apply. But I think we don't have to do that anymore. Like given today's progres…”
Yao: Lack of realistic benchmarks is AI's primary bottleneck
“So I think right now the problem is not even that we don't have good methodologies, it's more about we don't have good tasks.”
Shunyu Yao says academic AI research overcomplicates methods on simplistic tasks
“And I think in general, what people do in academia that I think is not good is they choose a very simple task, like Alford, and then they apply overly complex methods and to show the improved two percent I think like you should probably match, you know, the le…”
Yao: SWE-bench succeeded by balancing auto-grading, practicality, and scalability
“And I think part of the reason that Sweetbench is so popular now is it kind of hits the balance between these three dimensions, right? Easy to evaluate and being actually practical and being scalable.”
Shunyu Yao believes coding is the best application for AI agents
“Obviously coding is the best application for agents because it's all the gradable. It's super important. You can make everything like API or code action, right?”
Shawn Wang says Devin's breakthrough was Agent-Computer Interfaces, not advanced planning
“The planner is like actually pretty simple, but ACI. That they book through on.”
Yao: Reliable tool design accounts for 90% of agent performance
“I think making the tool good and reliable is probably like 90% of the whole agent. Once the tool is actually good, then the agent design can be much, much simpler. On the other hand, if the tool is bad, then no matter how much you put into the agent design pla…”
Harrison Chase says TypeScript yields better LLM tool-calling performance than JSON
“I saw some paper that used TypeScript notation instead of JSON notation for tool calling and it got a lot better performance.”
Shunyu Yao says agent interfaces should leverage large context over temporal steps
“If you look at find or whatever terminal command, you know, you can only look at one thing at a time, or that's because we have a very small working memory. You can only deal with one thing at a time. You can only look at one paragraph of text at the same time…”
Yao: Human cognition should serve as an AI reference point, not a blueprint
“I don't think we should copy exactly what's going on with human all the way, but I think it's good to have a reference point because this is a working example of how intelligence works.”
Yao: Improving AI agents requires better data, not architectural changes
“I think it's data. I think it's data because like changing architecture now is too hard and we don't have a good, better alternative solution now. I think it's mostly about data and agent data is obviously hard because People just write down the final result o…”
Shunyu Yao predicts training models on human computer trajectories achieves AGI
“The simplest way to achieve AGI is literally just record the re-actuatory of every human being and just put them together, you know, like what do you have thought about? What do you have done? Let's say on the computer, right? Imagine like solid experiment. Li…”
Chase: Few-shot prompting works better than detailed instructions for agent trajectories
“I'm pretty bullish on it, to be honest, for a few reasons. Like, one, I think it can maybe help for more complex things, but then also, two, like, it's a form of prompting, and prompting is just Communicating with the model what you want it to do. And sometime…”
Harrison Chase says developers should guide agent planning explicitly in code
“Sometimes I say that like the LLMs aren't
[4137] Great at planning yet.
[4138] So we can help them plan by telling them how to plan and code.
[4140] Cause that's very explicit and that's a good way of communicating how they should plan and stuff like that.”
Harrison Chase admits LangChain's memory service lacked product-market fit
“The memory service we launched, I don't think really found product market fit.”
Chase: LangGraph is adding cross-thread state persistence for agent memory
“Right now they're all persistent for a single thread.
[4392] We're going to add the ability to persist that between threads.
[4395] So then if you basically want to scope a memory to a user ID or to an assistant or to an organization, then you can do that.”
Yao: Enterprise customer support AI requires 99% reliability over simple tasks, not search
“It's very different from coding or web agent or whatever people are doing, because it's more about how can you do simple things reliably
It's not about, you know, can you sample a hundred times and you find one good mass proof or kill solution.
It's more about…”