Jul 11, 2026 · 48m · latent-space
Why AI Agents Don't Actually Understand You — Danielle Perszyk, Amazon AGI Lab
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Danielle Perszyk from the Amazon AGI Lab joins Swix to discuss why cognitive science, collective intelligence, and Theory of Mind are essential for building reliable AI agents. She outlines the lab's mission to escape narrow chatbot paradigms and develop foundational cognitive systems that augment human agency and flourishing.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 5.6 Guest teaching 5.4 Guest disagreement 1.8 The hosts pushing back 2.7
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 24:25 Rejecting convergence of generative video and reasoning
Danielle directly disputes Swix's suggestion that 3D world models and cognitive agents will fully converge, explaining that selective representation makes 1:1 simulation unworkable.
Hardest push from the hosts ▶ 34:05 Swix rejects bio-mimicry and agent autonomySwix strongly challenges Danielle's cognitive-science-first approach by invoking the bird-to-airplane analogy and rejecting the idea of autonomous agents with free will.
Biggest teaching moment ▶ 16:20 Reframing reliability from pixel clicks to mental modelsDanielle corrects Swix's mechanical coordinate-based view of reliability, educating him on how human tasks require continuously tracking unfolding user intentions.
The host holds their own ▶ 7:21 Swix catalogs real-time voice architecturesSwix demonstrates deep industry and research knowledge by citing Flamingo, Moshi, and Gradium as prior full-duplex architectures before OpenAI's GPT-4o.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Human Flourishing and Collective Intelligence at Adam Smith's House | 4 | 5 | 1 | 2 | Swix introduces Danielle with references to the Wealth of Nations anniversary in Edinburgh. Danielle articulates how human intelligence is inherently collective and social, advocating for AI designed around human flourishing rather than pure automation. | |
| Rethinking Workplace Automation and the Reality of AI Reliability | 3 | 5 | 2 | 1 | Swix asks whether people should worry about AI taking jobs. Danielle argues that current work over-indexes on screen drudgery, but cautions that existing agents remain too unreliable to achieve meaningful automation. | |
| Inside Amazon AGI Lab: Real-Time Interaction and Cognitive Memory | 7 | 6 | 2 | 3 | Danielle outlines Amazon AGI Lab's focus on real-time interactive perception agents and episodic memory. Swix demonstrates strong domain knowledge citing historical full-duplex systems like Flamingo, Moshi, and Gradium, and probes whether memory requires weight updates. | |
| Amazon AGI Lab's Startup Operating Model for Frontier Science | 5 | 4 | 2 | 1 | Danielle describes how the Adept team retained a startup operating model inside Amazon to focus on foundational science. Swix commiserates with her critique of other labs getting bottlenecked by short-term B2B SaaS demands. | |
| Redefining Reliability: From Pixel Coordinates to User Theory of Mind | 5 | 7 | 2 | 2 | Swix frames Nova Act as solving bounding box UI click coordinates. Danielle reframes reliability completely, explaining that real reliability requires modeling the user's unfolding intentions and mind rather than just clicking coordinates accurately. | |
| Beyond Next-Token Prediction: Representational Alignment as an Objective | 6 | 6 | 2 | 4 | Swix asks whether the field has objectives beyond next-token prediction. Danielle details representational alignment and Goodhart's law, while Swix questions whether current progress is an architectural limitation or merely a data problem. | |
| Agent Environments and the Social Nature of World Models | 7 | 6 | 2 | 4 | Swix probes the environment startup wave and asks whether generative 3D video world models converge with text reasoning. Danielle explains that human world models are fundamentally social, arguing true full convergence is conceptually flawed due to signal-to-noise dynamics. | |
| Escaping the Local Attractor State of Chatbots and Coding Agents | 5 | 5 | 2 | 2 | Swix asks about the lab's strategy between shipping products and publishing research. Danielle stresses avoiding premature productization that locks research into the local attractor of coding agents and chatbots. | |
| Cumulative Culture and Emergent Dynamics in Multi-Agent Systems | 7 | 6 | 2 | 3 | Danielle critiques current rigid multi-agent orchestration for lacking durable cumulative culture. Swix connects this to his interview with Noam Brown on cooperative/competitive dynamics and raises the limits of text-based shared memory. | |
| Preserving Human Agency: Marr's Levels and Mitigating Mode Collapse | 7 | 7 | 3 | 6 | Swix delivers sharp pushback using the bird vs airplane analogy to argue against bio-mimicry and free will for agents. Danielle responds by citing Marr's computational level of analysis, mode collapse in science, and aligning representations. | |
| Reinventing Education: Socratic AI and the Oxford Tutorial Model | 6 | 5 | 1 | 2 | Danielle proposes that interactive Socratic agents can prevent cognitive offloading, citing the Oxford tutorial model. Swix complements the concept by referencing Bloom's 2 Sigma problem in educational scaling. | |
| Eliminating Knowledge Work Friction and Looking Ahead to Season Two | 5 | 3 | 1 | 2 | Danielle previews Season 2 of Making a Mind. Swix grounds the episode in daily knowledge work bottlenecks like podcast video production, and Danielle agrees that eliminating digital drudgery remains the starting foundation. |
Statements from this episode (16)
Perszyk: Human intelligence is fundamentally collective and driven by interconnectivity
“The big idea is that human intelligence is collective. Anthropologists say that we've got the collective brain. No one individual is capable of even surviving on their own. We depend upon the collective. The intelligence emerges from our interactions. It's fun…”
Perszyk: An automation-only mindset leaves major AI value on the table
“The problem is that if we get the Trapped in this automation mindset, we are leaving on the table so much more value for what AI could be doing for not only human work, but human wellbeing, human interactions, human relationships.”
Perszyk: Current AI agents remain too unreliable to automate substantial work
“We look at what the metrics of the actual agents and they're so unreliable that ironically we feel a little bit better. The AI is actually not where we need it to be. To automate enough of the work.”
Perszyk: Amazon AGI Lab imported Adept's mission to automate any task
“Amazon AGI Lab is building human-aligned intelligence, and starting with AI that can do anything that a human can do on a computer, that was Adepth's original mission, and we kind of imported it and seeded the lab.”
Perszyk: AI industry is trapped in a batch turn-taking chatbot paradigm
“We're kind of trapped in this local attractor state of chatbots and coding agents and like turn taking in batches. And this is absolutely not how humans interact with each other.”
Perszyk: Adept negotiated an insulated startup model within Amazon AGI Lab
“Yeah, so I came with the original adept folks, and when we came, we convinced leadership that in order to do frontier level research, we really needed to keep an operating model that was more like a startup. We needed to insulate our research and be able to fo…”
Perszyk: Competing AI labs become victims of their own product success
“I think some of the other labs are, in a sense, victims of their own success because they have to Once they have a product out there that a lot of users are interacting with, they have to shut down different, you know, research projects and say, okay, all hand…”
Perszyk: Agent reliability requires modeling user intent over UI clicking
“So ultimately reliability has less to do with clicking in the same place and scrolling and more to do with modeling the user's mind. And that shift is everything that reframes how we think about what it is that we're building.”
Perszyk: Task-specific reinforcement learning fails to produce generalizable intelligence
“You can use things like reinforcement learning to get them really good at specific tasks that we might care about, but you do that for one task and you, it is not good at another task or it doesn't generalize.”
Perszyk: AI labs should spend as much on environments as compute
“We should be thinking about spending as much on the environments as on the compute and the data, because the environments literally shape the, what intelligence can emerge.”
Perszyk: Generative world models and reasoning models cannot entirely converge
“Unless I'm misunderstanding something, I don't think that they could entirely converge in principle because if an AI is generating exactly the same world that it exists in, The signal to noise ratio is, is non-existent. Like part of what makes us so flexible i…”
Perszyk: The AI industry will outgrow chatbot paradigms within months
“I imagine that. In a matter of months, we will look back at today and we won't even be able to empathize with the mental models that we have right now because they are so over-indexed on chatbots and coding agents.”
Perszyk: AI Industry Is in an Echo Chamber Building AI for Engineers
“Right now we're Building AI again for the people who are building AI. We're building AI for engineers, and we're all in our little echo chamber in the bay, and we're proud of ourselves for, you know, building AI.”
Perszyk: AI writing suggestions subconsciously shift users to opposing arguments
“There are studies that show that people will, even below their threshold of awareness, start with one argument and then be switched to a completely different, maybe opposing argument because of accepting all of these AI suggestions.”
Perszyk: AI tools boost individual output but narrow overall scientific research
“Individual scientists who are using AI tools are benefiting because they are producing more papers. They are getting more grants accepted. But science as a whole is narrowing.”
Perszyk: Mitigating AI Homogenization Requires a Diverse Society of Models
“The only way to counter that, and again, this is throughout human history, throughout human evolution, is to increase the diversity of ideas, the size of the ideas, the size of the population, and the interconnectivity of the ideas. So rather than having indiv…”