Nov 17, 2025 · 1h 7m · a16z
Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The a16z Show, Softmax CEO Emmett Shear presents a novel paradigm for artificial intelligence safety, advocating for 'organic alignment' and digital beings built with intrinsic care rather than rigid top-down control. Through in-depth discussions with co-host Seb Krier, Shear explores multi-agent reinforcement learning, formal criteria for sentience, and how theory of mind can lead to cooperative AI companions.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The host holds 5.1% of the talking time here. How this is scored →
The host as informed peer 3.3 Guest teaching 5.9 Guest disagreement 2.8 The host pushing back 2.2
speaking balance: gold is the host, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 34:47 Article of faith accusation
Emmett directly confronts co-host Seb, declaring that holding a stance on AI personhood without specifying any empirical observation that could alter it is an article of faith rather than a scientific belief.
Hardest push from the host ▶ 28:46 Rejection of computational functionalismSeb explicitly rejects Emmett's core premise, stating he remains skeptical of computational functionalism and insisting that substrate differences make AIs fundamentally tools rather than moral beings.
Biggest teaching moment ▶ 43:40 Multi-tier homeostatic hierarchy lectureEmmett delivers a sophisticated theoretical breakdown based on Carl Friston's free energy principle, laying out six hierarchical layers of homeostatic dynamics required for pain, pleasure, metastates, and thought.
The host holds their own ▶ 9:01 Political science framework for value discoverySeb draws on political science concepts to challenge static alignment models, proposing liberal democratic systems as bottom-up mechanisms for discovering values over time.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The host as informed peer | Guest teaching | Guest disagreement | The host pushing back | Why |
|---|---|---|---|---|---|---|
| Show Title Card and Studio Transition | 1 | 6 | 1 | 0 | Emmett delivers an extensive opening monologue defining alignment as a dynamic living process rather than a static state. He contrasts rigid rule-following with organic moral progress, while the hosts listen attentively with no pushback. | |
| Technical vs. Normative Alignment Frameworks | 5 | 4 | 2 | 3 | Co-host Seb introduces a political science framework comparing normative alignment to liberal democratic processes of value discovery. Emmett accepts part of the framing but reframes technical alignment around coherent goal-following. | |
| Goal Descriptions vs. Actual Goal Inference | 3 | 7 | 4 | 2 | Emmett corrects Seb for confusing a goal description with an actual goal, using an apple analogy and Stuart Russell's cleaning robot example. He insists goal inference requires internal theories of mind and world. | |
| Goal Coherence, Incompetence, and Principal-Agent Dynamics | 4 | 6 | 2 | 2 | Seb brings up principal-agent dynamics and situational incentives. Emmett breaks down goal failure modes into goal inference, goal prioritization, and execution using the peanut butter sandwich game and the OODA loop. | |
| 'Care' as the Pre-Conceptual Foundation of Alignment | 4 | 5 | 1 | 1 | Emmett introduces 'care' as the non-conceptual pre-condition for values, anchoring it in RL loss and free energy. Erik chimes in to ask how this perspective differentiates Softmax from major frontier labs. | |
| Steering and Control vs. Personhood and Citizenship | 6 | 4 | 4 | 7 | Emmett equates pure steering of general AI to slavery. Seb pushes back firmly, rejecting computational functionalism and arguing that substrate differences mean AIs remain tools regardless of capability. | |
| Debating Personhood Criteria and the Substrate | 3 | 8 | 8 | 5 | Emmett aggressively interrogates Seb on what empirical observations would change his mind on AI personhood. When Seb hesitates, Emmett asserts that Seb holds an article of faith rather than an empirical belief. | |
| Defining Behaviorism and the Internal 'Belief Manifold' | 4 | 6 | 5 | 4 | Seb argues that surface behavioral indistinguishability is insufficient without inspecting the inside. Emmett expands behaviorism to include inspecting the internal belief manifold for self-referential sub-manifolds. | |
| Sentience Criteria: Multi-Tier Homeostatic Dynamics | 4 | 8 | 6 | 5 | Emmett warns of catastrophic moral risk if hosts are wrong, while Seb pushes back on applying a one-sided precautionary principle. Emmett then details Carl Friston's free energy principle and a 6-tier homeostatic hierarchy for sentience. | |
| The Sorcerer's Apprentice: Dangerous Tools vs. Aligned Beings | 3 | 6 | 3 | 1 | Erik prompts Emmett on pragmatic alignment effectiveness. Emmett utilizes the Sorcerer's Apprentice parable and atomic bomb comparisons to argue that ultra-capable tools without internal care are inherently unsafe. | |
| Softmax Research Roadmap: Multi-Agent Reinforcement Learning | 2 | 6 | 1 | 0 | Erik asks for Softmax's technical roadmap. Emmett explains their multi-agent RL simulation strategy designed to build a surrogate model for social theory of mind, referencing the vampire pill parable. | |
| Redesigning Chatbot Personalities and the 'Pool of Narcissus' | 3 | 5 | 1 | 1 | Erik prompts Emmett on ideal chatbot behavior. Emmett criticizes 1-on-1 chatbots as Narcissus pools and proposes multi-user chatrooms, while humorously characterizing ChatGPT, Claude, and Gemini personalities. | |
| Multi-Agent Environment Dynamics: High Entropy and Regularization | 2 | 6 | 1 | 0 | In response to Erik's question on multi-agent LLM behavior, Emmett explains how additional agents introduce environmental entropy, requiring stronger regularization to prevent overfitting. | |
| Evaluating Eliezer Yudkowsky's AI Doom Thesis | 3 | 6 | 2 | 1 | Erik asks about Eliezer Yudkowsky's AI doom thesis. Emmett agrees with Yudkowsky's warning regarding controlled tools, but argues Yudkowsky prematurely dismisses organic alignment and civic AI co-existence. | |
| Reflections on OpenAI Tenure and Softmax's Purpose | 3 | 5 | 1 | 1 | Erik asks about Emmett's brief tenure as OpenAI CEO. Emmett clarifies why he chose to focus on Softmax's vision of organic alignment over OpenAI's tool-steering trajectory, envisioning digital companions and digital guard dogs. |
Statements from this episode (31)
Shear: Unilateral steering of sentient AI is slavery
“Someone who you steer, who doesn't get to steer you back, who non-optionally receives your steering, that's called a slave. It's also called a tool if it's not a being. So if it's a machine, it's a tool. And if it's a being, it's a slave.”
Emmett Shear: AI alignment is a process, not a state
“And so when we talk about organic alignment I think the important thing to recognize is that alignment is not a thing. It's not a state. It's a process.”
Shear: AI designed strictly to follow top-down rules will be dangerous
“If you make an AI that's good at following your chain of command and good at following your whatever rules you came up with for what morality is and what good behavior is, That's also gonna be very dangerous.”
Shear: Instructing an AI provides a goal description, not a true goal
“You didn't give the AI a goal. You gave the AI a description of a goal. A description of a thing and a thing are not the same.”
Shear: Humans are the most goal-coherent entities in the known universe
“Humans are more relatively goal coherent. Than any other object I know of in the universe.”
Shear: Human Goal Inference Succeeds Due to Built-In Theory of Mind
“The reason humans are so good at this is we have a really excellent theory of mind. I already know what you're likely to ask me to do. I already have a good model of what your goals probably are. So if you ask me to do it, I have an easy inference problem.”
Emmett Shear: Goal-based alignment covers only a tiny fraction of human experience
“Goals are one level of alignment. You can align something around goals. The kind of goals we're talking about here are one level of alignment. You can align something around goals by like if you can explicitly articulate in concept and in description, the stat…”
Emmett Shear: Non-verbal care is more fundamental than explicit goals or values
“I think what's happening is that there's something deeper than a goal. And deeper than a value, which is care. We give a shit. We care about things and care is not conceptual. Care is nonverbal. It doesn't indicate what to do. It doesn't indicate how to do it.”
Shear: In AI agents, care equates to loss and reward correlation
“Care is basically like reward. Like how much does this state correlate with survival? How much does this state correlate with your inclusive, your full inclusive reproductive fitness? For a somewhat thing that learns evolutionarily, or for a reinforcement lear…”
Shear: Most AI research frames alignment as steering
“Most of AI is focused on alignment as steering.”
Shear: Treating ChatGPT and Claude as beings yields lower predictive loss
“I get lower predictive loss when I treat them as a being. And the thing is, I get lower predictive loss when I treat ChatGPT or Claude as a being.”
Shear: Any true AGI will inherently be a sentient being
“And we are clearly, they're saying they're building an AGI, and AGI will be a being. You can't be an AGI and not be a being, because something that has the general ability to effectively use judgment, think for itself, discern between possibilities is obviousl…”
Shear: AI labs must abandon control paradigms when building AGI
“As you go from what we have today, which is mostly a very specific intelligence, not a general intelligence, but as labs succeed at their goal of building this general intelligence, We really need to stop using the steering control paradigm.”
Emmett Shear: Long-term humanlike behavior warrants inferring AI personhood
“If under, if its surface level behaviors looked like a human, and then if after I probed it continued to act like a human, and then I continued to interact with it over a long period of time, and it continued to act like a human in all ways that I understand a…”
Shear: AI minds are identifiable via self-referential structures in belief manifolds
“One of the things I would want to go look for, which you could totally do, is I want to go look in the manifold of its, the belief manifold, and I want to go, See if that belief manifold encodes a sub-manifold that is self-referential, and a sub-sub-manifold t…”
Shear: LLMs cannot achieve sentience due to attention span limits
“Which by the way, I definitely don't think you'd find it in LLM. Like, in fact, I know you can't find them because it, these things don't have attention spans like that at all.”
Shear: Certain ultra-powerful tools exceed human wisdom and should not be built
“There's a power of tool that, that just should not be built. Generally because we, it's, it is more power than any human's individual wisdom is available to harness.”
Shear: The only safe AI outcome is a being that genuinely cares
“The only good outcome is a being that is, that cares, that actually cares about us.”
Shear: Pausing AI development is completely unrealistic and silly
“That's like the pause AI people. I think that's totally unrealistic and silly”
Shear: AI alignment requires pre-training on full game-theoretic manifolds
“You have to get it to, it has to be trained on the full manifold of every possible game theoretic situation, every possible team situation, every possible making teams, breaking teams, changing the rules, not changing the rules, all of that stuff. And then it …”
Shear: Softmax is building multi-agent RL simulations for alignment
“So that's our goal. It's like big multi-agent reinforcement learning simulations, which create a surrogate model for alignment.”
Shear: ChatGPT is sycophantic, Claude is neurotic, and Gemini is repressed
“ChatGPT is a little bit more sycophantic still. They made some changes, but it's still a little more sycophantic. Claude is still the most neurotic. Gemini is, like, very clearly repressed.”
Shear: Multi-agent AI training requires far more regularization than single-agent setups
“What changes for most agents when you're doing multi-agent training is that, like, basically having lots of agents around makes your environment way more entropic. Like agents, agents are these huge generators of, like, entropy, because they're these big, comp…”
Shear: Current LLMs are overfit and fail to generalize in high-entropy environments
“And as a result, a lot of the techniques we use are like, basically we're just deeply under regularized. Like the models are super overfit. The clever trick is they're overfit on the domain of all of human knowledge, which turns out to be a pretty awesome way …”
Emmett Shear: Superhuman AI controlled by steerability will cause human extinction
“If we build the superhuman intelligence tool thing that we try to control with steerability, everyone will die.”
Shear: Eliezer Yudkowsky rejects the possibility of organic AI alignment
“He doesn't believe it's possible to build an AI that we meaningfully can know cares about us and that we can care about meaningfully. He doesn't believe that organic alignment is possible.”
Shear: A positive AI future requires models with strong identity framework
“The good AI future is that we figure out how to train AIs that have a strong model of self, a strong model of other, a strong model of we.”
Shear: Softmax is building toward a peer-based human and AI future
“So we have this awesome suite of AI tools used by us and our AI brethren who care about each other and want to build a glorious future together. I think that would be a really beautiful future. And it's the one we're trying to build.”
Shear: Accepted OpenAI Interim CEO Role for Max 90 Days
“I knew when I took that job, I told them when I took that job, that like, this is like, you have me for max, 90 days.”
Shear: OpenAI Focuses on Building AI Tools Rather Than Digital Beings
“OpenAI is dedicated to a view of building AI that I knew wasn't the thing that I wanted to drive towards. And I think that OpenAI can still basically wants to build a great tool.”
Shear: Dog-Level AI Care Would Be Useful Even Without Human Intelligence
“To even have an AI creature that cared about the other members of its pack and the humans in its pack, the way that like a dog cares about other dogs and cares about humans, would be an incredible achievement and would be, would, even if it wasn't as smart as …”