Opinion
Shear: Unilateral steering of sentient AI is slavery
“Someone who you steer, who doesn't get to steer you back, who non-optionally receives your steering, that's called a slave. It's also called a tool if it's not a being. So if it's a machine, it's a tool. And if it's a being, it's a slave.”
Prediction Not checkable as stated
Shear: AI designed strictly to follow top-down rules will be dangerous
“If you make an AI that's good at following your chain of command and good at following your whatever rules you came up with for what morality is and what good behavior is, That's also gonna be very dangerous.”
Assertion Not checkable as stated
Shear: Treating ChatGPT and Claude as beings yields lower predictive loss
“I get lower predictive loss when I treat them as a being. And the thing is, I get lower predictive loss when I treat ChatGPT or Claude as a being.”
Prediction Not checkable as stated
Shear: Any true AGI will inherently be a sentient being
“And we are clearly, they're saying they're building an AGI, and AGI will be a being. You can't be an AGI and not be a being, because something that has the general ability to effectively use judgment, think for itself, discern between possibilities is obviousl…”
Insight
Shear: AI labs must abandon control paradigms when building AGI
“As you go from what we have today, which is mostly a very specific intelligence, not a general intelligence, but as labs succeed at their goal of building this general intelligence, We really need to stop using the steering control paradigm.”
Prediction Not checkable as stated
Emmett Shear: Superhuman AI controlled by steerability will cause human extinction
“If we build the superhuman intelligence tool thing that we try to control with steerability, everyone will die.”
Opinion
Shear: The only safe AI outcome is a being that genuinely cares
“The only good outcome is a being that is, that cares, that actually cares about us.”
Insight
Shear: Instructing an AI provides a goal description, not a true goal
“You didn't give the AI a goal. You gave the AI a description of a goal. A description of a thing and a thing are not the same.”
Insight
Emmett Shear: Non-verbal care is more fundamental than explicit goals or values
“I think what's happening is that there's something deeper than a goal. And deeper than a value, which is care. We give a shit. We care about things and care is not conceptual. Care is nonverbal. It doesn't indicate what to do. It doesn't indicate how to do it.”
Insight
Shear: In AI agents, care equates to loss and reward correlation
“Care is basically like reward. Like how much does this state correlate with survival? How much does this state correlate with your inclusive, your full inclusive reproductive fitness? For a somewhat thing that learns evolutionarily, or for a reinforcement lear…”
Insight
Shear: AI minds are identifiable via self-referential structures in belief manifolds
“One of the things I would want to go look for, which you could totally do, is I want to go look in the manifold of its, the belief manifold, and I want to go, See if that belief manifold encodes a sub-manifold that is self-referential, and a sub-sub-manifold t…”
Assertion Not checkable as stated
Shear: LLMs cannot achieve sentience due to attention span limits
“Which by the way, I definitely don't think you'd find it in LLM. Like, in fact, I know you can't find them because it, these things don't have attention spans like that at all.”
Insight
Shear: Certain ultra-powerful tools exceed human wisdom and should not be built
“There's a power of tool that, that just should not be built. Generally because we, it's, it is more power than any human's individual wisdom is available to harness.”
Insight
Shear: AI alignment requires pre-training on full game-theoretic manifolds
“You have to get it to, it has to be trained on the full manifold of every possible game theoretic situation, every possible team situation, every possible making teams, breaking teams, changing the rules, not changing the rules, all of that stuff. And then it …”
Opinion
Shear: ChatGPT is sycophantic, Claude is neurotic, and Gemini is repressed
“ChatGPT is a little bit more sycophantic still. They made some changes, but it's still a little more sycophantic. Claude is still the most neurotic. Gemini is, like, very clearly repressed.”
Insight
Shear: Current LLMs are overfit and fail to generalize in high-entropy environments
“And as a result, a lot of the techniques we use are like, basically we're just deeply under regularized. Like the models are super overfit. The clever trick is they're overfit on the domain of all of human knowledge, which turns out to be a pretty awesome way …”
Prediction Not checkable as stated
Shear: Softmax is building toward a peer-based human and AI future
“So we have this awesome suite of AI tools used by us and our AI brethren who care about each other and want to build a glorious future together. I think that would be a really beautiful future. And it's the one we're trying to build.”
Opinion
Shear: OpenAI Focuses on Building AI Tools Rather Than Digital Beings
“OpenAI is dedicated to a view of building AI that I knew wasn't the thing that I wanted to drive towards. And I think that OpenAI can still basically wants to build a great tool.”
Insight
Emmett Shear: AI alignment is a process, not a state
“And so when we talk about organic alignment I think the important thing to recognize is that alignment is not a thing. It's not a state. It's a process.”
Insight
Shear: Humans are the most goal-coherent entities in the known universe
“Humans are more relatively goal coherent. Than any other object I know of in the universe.”
Insight
Emmett Shear: Goal-based alignment covers only a tiny fraction of human experience
“Goals are one level of alignment. You can align something around goals. The kind of goals we're talking about here are one level of alignment. You can align something around goals by like if you can explicitly articulate in concept and in description, the stat…”
Insight
Emmett Shear: Long-term humanlike behavior warrants inferring AI personhood
“If under, if its surface level behaviors looked like a human, and then if after I probed it continued to act like a human, and then I continued to interact with it over a long period of time, and it continued to act like a human in all ways that I understand a…”
Opinion
Shear: Pausing AI development is completely unrealistic and silly
“That's like the pause AI people. I think that's totally unrealistic and silly”
Prediction Open · timeframe Nov 2028
Shear: Softmax is building multi-agent RL simulations for alignment
“So that's our goal. It's like big multi-agent reinforcement learning simulations, which create a surrogate model for alignment.”
Emmett Shear