Qiu: Pure reinforcement learning cannot deliver planning and reasoning
Kanjun Qiu · Why AI Agents Don't Work (yet) - with Kanjun Qiu of Imbue · Oct 21, 2023 · at 25:16
Kanjun Qiu, CEO of Imbue, explains why Imbue shifted from end-to-end reinforcement learning simulation environments toward language-based foundation models.
0:00 / 0:07exact quote · 7.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The second thing we learned is that reinforcement learning is not a good vehicle. Like, pure reinforcement learning is not a good vehicle for planning and reasoning.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Kanjun Qiu
Qiu: Reasoning is the single biggest blocker for AI agents
“Reasoning is actually, we believe the biggest blocker to agents or systems that can do these larger goals.”
Qiu: Models lack reasoning because the internet lacks explicit reasoning data
“And models today, they're not optimized for reasoning. It turns out that there's not actually that much explicit reasoning data on the internet.”
Qiu: Synthetic code data can improve reasoning better than human code
“Code also is a big piece of improving reasoning. So yeah generated code is not That much worse than, like, regular human written code. You might even say it could be better in a lot of ways.”
Kanjun Qiu: Most autonomous AI agents do not work well today
“So to your question of, like, what agents work well and what doesn't work well, like, most of the agents don't work well, and we're slowly making them work better by improving the underlying model and improving these.”
Qiu: Imbue generates specific reasoning data rather than relying on web data
“So I think internally, yeah, we have a lot of thoughts on what reasoning is, and we generate a lot more specific data. We're not just like, oh, it'll figure out reasoning from this black box or like, it'll figure out reasoning from the data that, that exists.”
Kanjun Qiu: Chat is a skeuomorphic, primitive interface for AI agents
“Chat as an interface is skeuomorphic. So in the early days, when we made word processors on our computers, they had notepad lines because that's what we understood you know, these like objects to be chat. Like texting someone is something we understand. So tex…”