Oct 21, 2023 · 1h 12m · latent-space
Why AI Agents Don't Work (yet) - with Kanjun Qiu of Imbue
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Imbue CEO Kanjun Qiu examines why current autonomous AI agents encounter severe reliability bottlenecks, detailing how foundation models optimized for explicit natural language reasoning, synthetic code data, and robust software abstractions will transform personal computing.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 25.3% of the talking time here. How this is scored →
The hosts as informed peer 5.4 Guest teaching 3.8 Guest disagreement 1.7 The hosts pushing back 2.8
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 32:45 Kanjun firmly rejects George Hotz's scale-solves-reasoning thesis
Kanjun directly counters the premise introduced from George Hotz, firmly stating that reasoning cannot be acquired by merely throwing massive data and compute at black boxes.
Hardest push from the hosts ▶ 49:49 Swyx directly presses Kanjun on Imbue versus AdeptSwyx asks an unvarnished competitive question, pushing Kanjun to justify why top talent should join Imbue instead of its most direct venture-backed peer Adept.
Biggest teaching moment ▶ 24:35 Kanjun details the structural failure modes of pure reinforcement learningKanjun provides a clear tutorial on why reinforcement learning algorithms fail completely without explicit curricula and lack the capacity for high-level planning.
The host holds their own ▶ 21:09 Swyx challenges synthetic data on distribution resampling biasSwyx brings rigorous machine learning skepticism, contrasting foundation model spending profiles and questioning the statistical validity of training models on synthetic model outputs.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Early Startups and Lessons from Sorceress AI | 5 | 3 | 1 | 2 | Swyx draws a sharp analogy between Sorceress's recruiting business and the dating app business model of endogenous churn. Kanjun validates the comparison and explains how trust barriers informed Imbue's interface approach. | |
| Founding Generally Intelligent and Free Intellectual Energy | 4 | 4 | 1 | 1 | Alessio asks about starting Generally Intelligent in 2021 before agents were mainstream. Kanjun explains her foundational thesis on free intellectual energy analogous to the Industrial Revolution and early self-supervised learning breakthroughs. | |
| Reasoning Models and Personal Computing Vision | 5 | 3 | 1 | 2 | Alessio and Swyx explore Imbue's focus on reasoning models and ask how they navigate between a pure research lab and a product company. Kanjun draws parallels to Apple's long multi-touch development cycles. | |
| Imbue's Internal Culture and Engineering Structure | 4 | 3 | 1 | 1 | Alessio queries how Imbue manages its unique dual culture after raising 200 million dollars. Kanjun outlines her social process philosophy of treating employees as creative agents rather than disposable corporate assets. | |
| Optimizing Pre-Training and Synthetic Data Generation | 7 | 4 | 2 | 5 | Swyx brings strong domain depth regarding compute versus data spending in foundation models, explicitly questioning whether synthetic model data causes distribution collapse. Kanjun explains that Imbue intentionally wants a spiky reasoning distribution rather than matching web text. | |
| Limits of Reinforcement Learning and Avalon Simulation | 5 | 6 | 2 | 2 | Alessio asks about the Avalon simulation benchmark, playfully prompting Kanjun to correct a quote. Kanjun delivers a thorough technical breakdown of why pure reinforcement learning without curricula fails at planning and why reasoning must happen in language. | |
| Current Agent Limitations and Architectural Abstractions | 5 | 4 | 1 | 3 | Swyx presses on whether simple single-task agents face an asymptote ceiling. Kanjun readily agrees, acknowledging that current agent abstractions are severely leaky and that pile-of-hacks solutions fail production reliability bars. | |
| Defining Reasoning and Natural Language Programming | 6 | 5 | 4 | 4 | Alessio references George Hotz's claim that sufficient compute will automatically emerge reasoning from human data. Kanjun explicitly rejects that premise, arguing that human reasoning strategies must be generated intentionally and that code serves as a structured curriculum. | |
| Interface Paradigms, Agent Protocols, and Evaluation | 7 | 3 | 2 | 3 | The hosts and Kanjun discuss skeuomorphic chat interfaces, historical computing transitions, agent communication protocols, and recent research like MetaGPT and Voyager. Both sides trade specific literature references comfortably. | |
| Context Memory Limitations and Imbue's Positioning | 7 | 4 | 2 | 6 | Swyx pushes hard with direct competitive questions, asking why RAG is insufficient for complex tasks and confronting Kanjun on why candidates should pick Imbue over Adept. Kanjun provides a structured defense centered on non-leaky abstractions and dogfooding. | |
| Curiosity, Online Learning, and Cultivating Scenius | 6 | 3 | 1 | 2 | Alessio and Swyx explore Kanjun's philosophical website questions and deep involvement in building hacker house ecosystems like The Archive and South Park Commons. Kanjun and Swyx discuss the concept of scenius and idea flux. | |
| Lightning Round, AI Governance, and Safety Takeaways | 4 | 3 | 2 | 2 | In the lightning round, Kanjun deflects Swyx's hypothetical alternative company question by asserting Imbue is the only thing worth building. She concludes with Imbue's multi-pillar approach to AI safety and empirical policy work with the Department of Commerce. |
Statements from this episode (20)
Qiu: Recruiting software suffers high churn because success halts hiring
“Recruiting is a market that is endogenously high churn. Which means, because people start hiring, and then we hire the role for them, and they stop hiring.”
Qiu: AI will provide free intellectual energy like the Industrial Revolution
“And what's happened over the last 150 years since the Industrial Revolution is we've kind of gotten free energy. Like, energy is way more free than it was a 150 years ago. And so, as a result, we've built all these technologies, like the stove, and the dishwas…”
Qiu: Reasoning is the single biggest blocker for AI agents
“Reasoning is actually, we believe the biggest blocker to agents or systems that can do these larger goals.”
Qiu: Models lack reasoning because the internet lacks explicit reasoning data
“And models today, they're not optimized for reasoning. It turns out that there's not actually that much explicit reasoning data on the internet.”
Imbue's CARBS hyperparameter optimizer originated from plasma physics concepts
“CARBS, our hyperparameter optimizer, came from Abe trying to automate his own research process doing hyperparameter optimization, and he actually pulled some ideas from plasma physics, he's a plasma physicist, to make the local search work.”
Kanjun Qiu: Agent reliability's final 20% is as hard as self-driving cars
“Agents haven't been productized yet for, partly for this reason, is that, like, the abstractions are very leaky. You know, we can get, like, 80% of the way there, but, like, self-driving cars, like, the remaining 20% is actually really difficult.”
Qiu: Synthetic code data can improve reasoning better than human code
“Code also is a big piece of improving reasoning. So yeah generated code is not That much worse than, like, regular human written code. You might even say it could be better in a lot of ways.”
Qiu: RL algorithms do not work at all without a curriculum
“One is with no curriculum, RL algorithms don't work at all.”
Qiu: Pure reinforcement learning cannot deliver planning and reasoning
“The second thing we learned is that reinforcement learning is not a good vehicle. Like, pure reinforcement learning is not a good vehicle for planning and reasoning.”
Qiu: Language reasoning in agents is primarily for human inspectability
“I personally think of it as it's much more important for us, the human user. So I think you probably could get end to end agents that work and are fairly general at some point in the future. But I think you don't want that. Like, we actually want agents that w…”
Kanjun Qiu: Most autonomous AI agents do not work well today
“So to your question of, like, what agents work well and what doesn't work well, like, most of the agents don't work well, and we're slowly making them work better by improving the underlying model and improving these.”
Kanjun Qiu: Agent development will evolve beyond bare metal to higher abstractions
“And I think it's basically a similar route here where we're like in the like bare metal phase of agent building, and we will eventually get to something with much nicer abstractions.”
Qiu: Imbue generates specific reasoning data rather than relying on web data
“So I think internally, yeah, we have a lot of thoughts on what reasoning is, and we generate a lot more specific data. We're not just like, oh, it'll figure out reasoning from this black box or like, it'll figure out reasoning from the data that, that exists.”
Qiu: Code is the most explicit reasoning curriculum for AI models
“Code is the most explicit example of reasoning data on the internet. Yeah. And it's not only structured, it's actually very explicit, which is nice. You know, it says this variable means this and then it uses this variable and then the function does this. Like…”
Kanjun Qiu: Chat is a skeuomorphic, primitive interface for AI agents
“Chat as an interface is skeuomorphic. So in the early days, when we made word processors on our computers, they had notepad lines because that's what we understood you know, these like objects to be chat. Like texting someone is something we understand. So tex…”
Kanjun Qiu: Standardizing agent protocols is premature because agents don't work yet
“Part of why I think it's early is because the issue with agents is it's not quite like the internet where you could like make a website and the website would appear. The issue with agents is that they don't work. And so it may be a bit early to figure out what…”
Qiu: Memory limitations block AI agents from handling complex, long-running tasks
“I think what we'll see is we'll get like relatively simplistic agents pretty soon, and they will get more and more complex. And there's like a future wave in which they are able to do these like really difficult, really long running tasks. And the blocker to t…”
Qiu: RAG is inadequate for scientific AI reasoning and cumulative synthesis
“I don't think RAG is enough for that kind of thing. But RAG is certainly enough for, like, user preferences and things like that.”
Kanjun Qiu: Larger models fine-tune faster and with higher sample efficiency
“As models get bigger, they fine tune faster. So they're more sample efficient as they get bigger.”
Imbue built an AI agent to analyze 20,000 Commerce Department policy proposals
“We built an agent that helped us analyze the, like, 20,000 pages of policy proposals submitted to the Department of Commerce request for AI policy proposals, and we, like, looked at what were the problems people brought up, and what were the solutions they pre…”