Guu: AI Agents Relying Purely on Explicit Reasoning May Fail to Scale
Kelvin Guu · No Priors Ep. 15 | With Kelvin Guu, Staff Research Scientist, Google Brain · May 4, 2023 · at 20:39
Google Brain researcher Kelvin Guu discusses architectural bottlenecks in current LLM-based autonomous agent designs with Sarah Guo and Elad Gil.
0:00 / 0:08exact quote · 8.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“That's something that seems to be missing from these autonomous agents right now. Everything is explicit reasoning, and at some point that might not scale or that might become brittle.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Kelvin Guu
Guu: Without Attribution, Distinguishing LLM Generalization From Memorization Is Impossible
“Without being able to track that down, we'll never know if these models are generalizing or just kind of cleverly patching together what they know.”
Guu: Fundamental AI Advances May Make Agent Workflow Engineering Obsolete
“But it's quite possible that fundamental advances could come along and this kind of workflow hacking or workflow engineering could go the same way as feature engineering or specialized architectures.”
Guu: FLAN Demonstrated Zero-Shot Generalization After Multi-Task Training on 100 Tasks
“What we were able to show was that if you train on a hundred tasks, if we then show the model a new 101 task, it will be able to adapt to that without having seen that particular task before.”
Guu: Tool-Using AI Models Still Struggle to Fulfill Original Retrieval Promises
“And since then we've encountered many interesting challenges on top of that idea that I would say even in, in the sort of systems you see today, these tool using models that issue Google searches or provide citations, they still face some challenges in terms o…”
Guu: Retrieval Models Best Serve Enterprise Needs Through Modularity and Privacy
“So it's actually been a case where I think more and more the benefits of a retrieval augmented model are for modularity. For personal information that you might not want in the main model, for adapting to, say, like an enterprise customer that has special info…”
Guu: Adapting LLMs to Code Requires Specialized Training Over Pure Retrieval
“So if you just want to get very precise factoid information, like what is my Wi-Fi password, retrieval augmentation is going to be very good. But if, for example, you're trying to adapt a language model to a new enterprise, and they have some kind of a special…”