Yao: Evaluator quality is the key bottleneck for agent self-reflection
Shunyu Yao · Language Agents: From Reasoning to Acting — with Shunyu Yao of OpenAI, Harrison Chase of LangGraph · Sep 27, 2024 · at 17:56
Shunyu Yao explains why self-reflection techniques succeed in coding tasks with deterministic error outputs but struggle in pure reasoning domains like math.
0:00 / 0:40exact quote · 40.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think a key bottleneck is the evaluator, right? Basically you need to have a good sense of the signal. So for example, like if you are trying to do a very hard reasoning task, say mathematics, For example, and you don't have any tools, right? It's operating in this chain of thought setup. Then reflection will be pretty hard because in order to reflect upon your thoughts, you have to have a very good evaluator to judge whether your thought is good or not. But that might be as hard as solving the problem itself or even harder. The principle of self-reflection is probably more applicable if you have a good evaluator, for example, in the case of like coding, right? Like if you have those arrows, then you can just reflect on that and How to solve the bug and stuff.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Shunyu Yao
Shunyu Yao predicts training models on human computer trajectories achieves AGI
“The simplest way to achieve AGI is literally just record the re-actuatory of every human being and just put them together, you know, like what do you have thought about? What do you have done? Let's say on the computer, right? Imagine like solid experiment. Li…”
Shunyu Yao says Ilya Sutskever claimed GPT-1 had solved language
“Back in OpenAI, they did this GPT-ONE together, and Ilya just said, Karthik, you should stay, because we just solved the language.”
Shunyu Yao advises developers to default to minimalist prompting for AI agents
“And I think in terms of the actual prompting method to use for a particular problem, I'm I think we should all be in the minimum list kind of camp, right? You should try the minimum thing and see if it works and if it doesn't work and there's absolute reason t…”
Shunyu Yao argues modern LLMs make prompt engineering tricks obsolete
“I feel like in some sense, I feel like prompt engineering, even it's like a slightly negative word at the time, because it refers to all those kind of weird tricks that you have to apply. But I think we don't have to do that anymore. Like given today's progres…”