Everything Emmett Shear said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Shear: Unilateral steering of sentient AI is slavery
“Someone who you steer, who doesn't get to steer you back, who non-optionally receives your steering, that's called a slave. It's also called a tool if it's not a being. So if it's a machine, it's a tool. And if it's a being, it's a slave.”
Shear: AI designed strictly to follow top-down rules will be dangerous
“If you make an AI that's good at following your chain of command and good at following your whatever rules you came up with for what morality is and what good behavior is, That's also gonna be very dangerous.”
Shear: Treating ChatGPT and Claude as beings yields lower predictive loss
“I get lower predictive loss when I treat them as a being. And the thing is, I get lower predictive loss when I treat ChatGPT or Claude as a being.”
Shear: Any true AGI will inherently be a sentient being
“And we are clearly, they're saying they're building an AGI, and AGI will be a being. You can't be an AGI and not be a being, because something that has the general ability to effectively use judgment, think for itself, discern between possibilities is obviousl…”
Shear: AI labs must abandon control paradigms when building AGI
“As you go from what we have today, which is mostly a very specific intelligence, not a general intelligence, but as labs succeed at their goal of building this general intelligence, We really need to stop using the steering control paradigm.”
Emmett Shear: Superhuman AI controlled by steerability will cause human extinction
“If we build the superhuman intelligence tool thing that we try to control with steerability, everyone will die.”
Shear: The only safe AI outcome is a being that genuinely cares
“The only good outcome is a being that is, that cares, that actually cares about us.”
Shear: Instructing an AI provides a goal description, not a true goal
“You didn't give the AI a goal. You gave the AI a description of a goal. A description of a thing and a thing are not the same.”
Emmett Shear: Non-verbal care is more fundamental than explicit goals or values
“I think what's happening is that there's something deeper than a goal. And deeper than a value, which is care. We give a shit. We care about things and care is not conceptual. Care is nonverbal. It doesn't indicate what to do. It doesn't indicate how to do it.”
Shear: In AI agents, care equates to loss and reward correlation
“Care is basically like reward. Like how much does this state correlate with survival? How much does this state correlate with your inclusive, your full inclusive reproductive fitness? For a somewhat thing that learns evolutionarily, or for a reinforcement lear…”
Shear: AI minds are identifiable via self-referential structures in belief manifolds
“One of the things I would want to go look for, which you could totally do, is I want to go look in the manifold of its, the belief manifold, and I want to go, See if that belief manifold encodes a sub-manifold that is self-referential, and a sub-sub-manifold t…”
Shear: LLMs cannot achieve sentience due to attention span limits
“Which by the way, I definitely don't think you'd find it in LLM. Like, in fact, I know you can't find them because it, these things don't have attention spans like that at all.”
Shear: Certain ultra-powerful tools exceed human wisdom and should not be built
“There's a power of tool that, that just should not be built. Generally because we, it's, it is more power than any human's individual wisdom is available to harness.”
Shear: AI alignment requires pre-training on full game-theoretic manifolds
“You have to get it to, it has to be trained on the full manifold of every possible game theoretic situation, every possible team situation, every possible making teams, breaking teams, changing the rules, not changing the rules, all of that stuff. And then it …”
Shear: ChatGPT is sycophantic, Claude is neurotic, and Gemini is repressed
“ChatGPT is a little bit more sycophantic still. They made some changes, but it's still a little more sycophantic. Claude is still the most neurotic. Gemini is, like, very clearly repressed.”
Shear: Current LLMs are overfit and fail to generalize in high-entropy environments
“And as a result, a lot of the techniques we use are like, basically we're just deeply under regularized. Like the models are super overfit. The clever trick is they're overfit on the domain of all of human knowledge, which turns out to be a pretty awesome way …”
Shear: Softmax is building toward a peer-based human and AI future
“So we have this awesome suite of AI tools used by us and our AI brethren who care about each other and want to build a glorious future together. I think that would be a really beautiful future. And it's the one we're trying to build.”
Shear: OpenAI Focuses on Building AI Tools Rather Than Digital Beings
“OpenAI is dedicated to a view of building AI that I knew wasn't the thing that I wanted to drive towards. And I think that OpenAI can still basically wants to build a great tool.”
Shear: Org structure has only two moves: centralizing and decentralizing
“There is a fundamental tension or trade-off between centralizing and decentralizing, and those are really the only two moves.”
Shear: Growing companies inevitably require reorgs every 18 to 36 months
“You just sort of inevitably wind up on a 18, 24 month, 36 month beat, having a reorg at some level. And that's not something to fear, but something that you should embrace.”
Shear: Communities cannot be built directly, they must be grown like gardens
“The particular object level advice would be that you don't build communities, you grow them. Your communities are more like a garden than they are a building. And like a garden, it's important that you, there's things you can do to make the garden more likely …”
Emmett Shear: AI alignment is a process, not a state
“And so when we talk about organic alignment I think the important thing to recognize is that alignment is not a thing. It's not a state. It's a process.”
Shear: Humans are the most goal-coherent entities in the known universe
“Humans are more relatively goal coherent. Than any other object I know of in the universe.”
Emmett Shear: Goal-based alignment covers only a tiny fraction of human experience
“Goals are one level of alignment. You can align something around goals. The kind of goals we're talking about here are one level of alignment. You can align something around goals by like if you can explicitly articulate in concept and in description, the stat…”