The Ledger, every show
Every statement that passed quotation and attribution checks, across all 44 shows. Pick shows below, then mix any filter with any other.
shows 
every show 44 of 44
Shunyu Yao predicts training models on human computer trajectories achieves AGI
“The simplest way to achieve AGI is literally just record the re-actuatory of every human being and just put them together, you know, like what do you have thought about? What do you have done? Let's say on the computer, right? Imagine like solid experiment. Li…”
Shunyu Yao says Ilya Sutskever claimed GPT-1 had solved language
“Back in OpenAI, they did this GPT-ONE together, and Ilya just said, Karthik, you should stay, because we just solved the language.”
Shunyu Yao advises developers to default to minimalist prompting for AI agents
“And I think in terms of the actual prompting method to use for a particular problem, I'm I think we should all be in the minimum list kind of camp, right? You should try the minimum thing and see if it works and if it doesn't work and there's absolute reason t…”
Shunyu Yao argues modern LLMs make prompt engineering tricks obsolete
“I feel like in some sense, I feel like prompt engineering, even it's like a slightly negative word at the time, because it refers to all those kind of weird tricks that you have to apply. But I think we don't have to do that anymore. Like given today's progres…”
Yao: Lack of realistic benchmarks is AI's primary bottleneck
“So I think right now the problem is not even that we don't have good methodologies, it's more about we don't have good tasks.”
Shunyu Yao believes coding is the best application for AI agents
“Obviously coding is the best application for agents because it's all the gradable. It's super important. You can make everything like API or code action, right?”
Yao: Reliable tool design accounts for 90% of agent performance
“I think making the tool good and reliable is probably like 90% of the whole agent. Once the tool is actually good, then the agent design can be much, much simpler. On the other hand, if the tool is bad, then no matter how much you put into the agent design pla…”
Yao: Pairing reasoning with tool use is essential for unfamiliar tools
“And I think the second contribution is this idea of what people call like inner monologue or thinking or reasoning or whatever to be paired with tool use. I think that's still not trivial because if you look at the default function calling or whatever, like th…”
Yao: Reflexion replaces scalar RL rewards with verbal gradient descent
“I think one way to think of reflection is that the traditional idea of reinforcement learning is you have a scalar reward, and then you somehow back propagate the signal of the scalar reward. To the rest of your neural network through whatever algorithm, like …”
Yao: Evaluator quality is the key bottleneck for agent self-reflection
“I think a key bottleneck is the evaluator, right? Basically you need to have a good sense of the signal. So for example, like if you are trying to do a very hard reasoning task, say mathematics, For example, and you don't have any tools, right? It's operating …”
Shunyu Yao says academic AI research overcomplicates methods on simplistic tasks
“And I think in general, what people do in academia that I think is not good is they choose a very simple task, like Alford, and then they apply overly complex methods and to show the improved two percent I think like you should probably match, you know, the le…”
Shunyu Yao says agent interfaces should leverage large context over temporal steps
“If you look at find or whatever terminal command, you know, you can only look at one thing at a time, or that's because we have a very small working memory. You can only deal with one thing at a time. You can only look at one paragraph of text at the same time…”
Yao: Human cognition should serve as an AI reference point, not a blueprint
“I don't think we should copy exactly what's going on with human all the way, but I think it's good to have a reference point because this is a working example of how intelligence works.”
Yao: Improving AI agents requires better data, not architectural changes
“I think it's data. I think it's data because like changing architecture now is too hard and we don't have a good, better alternative solution now. I think it's mostly about data and agent data is obviously hard because People just write down the final result o…”
Shunyu Yao built the ReAct prototype before Chain-of-Thought existed
“The prototype I think was around November of 2021. So that's even before like chain of thought or whatever came up.”
Yao: Text adventure games remain very hard even for GPT-4
“Like those texting are just too hard. I think today it's still very hard. Like if you used to be before to solve it, it's still very hard.”
Yao: SWE-bench succeeded by balancing auto-grading, practicality, and scalability
“And I think part of the reason that Sweetbench is so popular now is it kind of hits the balance between these three dimensions, right? Easy to evaluate and being actually practical and being scalable.”
Yao: Enterprise customer support AI requires 99% reliability over simple tasks, not search
“It's very different from coding or web agent or whatever people are doing, because it's more about how can you do simple things reliably
It's not about, you know, can you sample a hundred times and you find one good mass proof or kill solution.
It's more about…”