Yao: Reflexion replaces scalar RL rewards with verbal gradient descent
Shunyu Yao · Language Agents: From Reasoning to Acting — with Shunyu Yao of OpenAI, Harrison Chase of LangGraph · Sep 27, 2024 · at 15:35
Shunyu Yao, AI researcher at OpenAI and co-creator of ReAct, explains the mental model behind the Reflexion agent architecture.
0:00 / 1:09exact quote · 69.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think one way to think of reflection is that the traditional idea of reinforcement learning is you have a scalar reward, and then you somehow back propagate the signal of the scalar reward. To the rest of your neural network through whatever algorithm, like policy gradient or A to C or whatever. And if you think about the real life, you know, most of the reward signal is not scalar. It's like your boss told you know, you should have done a better job in this, but a good job on that or whatever, right? It's not like a scalar reward, like 29 or something. I think in general, human do more deal more with, you know, long scalar reward, or you can say language feedback, right? And the way That they deal with language feedback also have this kind of back propagation kind of process, right? Because you start from this, you did a good job on job B and then you reflect, you know, what could have done different to change, to make it better. And you kind of change your prompt, right? Basically you change your prompt on how to do job A and how to do job B. And then you do the whole thing again. So it's really like a pipeline of language where it's self-created descent. You have something like tax reasoning to replace Those gradient descent algorithms.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Shunyu Yao
Shunyu Yao predicts training models on human computer trajectories achieves AGI
“The simplest way to achieve AGI is literally just record the re-actuatory of every human being and just put them together, you know, like what do you have thought about? What do you have done? Let's say on the computer, right? Imagine like solid experiment. Li…”
Shunyu Yao says Ilya Sutskever claimed GPT-1 had solved language
“Back in OpenAI, they did this GPT-ONE together, and Ilya just said, Karthik, you should stay, because we just solved the language.”
Shunyu Yao advises developers to default to minimalist prompting for AI agents
“And I think in terms of the actual prompting method to use for a particular problem, I'm I think we should all be in the minimum list kind of camp, right? You should try the minimum thing and see if it works and if it doesn't work and there's absolute reason t…”
Shunyu Yao argues modern LLMs make prompt engineering tricks obsolete
“I feel like in some sense, I feel like prompt engineering, even it's like a slightly negative word at the time, because it refers to all those kind of weird tricks that you have to apply. But I think we don't have to do that anymore. Like given today's progres…”