The Exchanges

Q Two last questions. We'll get you out of here in as much detail as possible. Can you explain your, what your vision of an AI future actually looks like, like a good, good AI future?

A Yeah, um, the good AI future is that we, we figure out how to train AIs that have a strong model of self, a strong model of other, a strong model of we. They, they know, they know about we's in addition to I's and you's, um, and they, they have a really strong theory of mind, and they care about other agents like them. Much the way that humans would, if you knew that that AI had experiences like you, and like you would extend You would care about those experiences, not infinitely, but you would. It, it does the exact same thing back to us. It's learned the same thing we've learned that like everything that lives and knows itself and that wants to live and wants to thrive is deserving of an opportunity to do so. And we are that, and it correctly infers that we are, and we live in a society where they are our peers and we care about them and they care about us and they're good teammates and they're good citizens and they're They're good parts of our society. Um, like, we're good parts of our society, which is to say, and to a finite, limited degree, where some of them turn into criminals and bad people and all that kind of stuff, and we have an AI police force that tracks down the bad ones, and, you know, same, and same as for everybody else, um, and that's, that's, that's what a good, that's what a good future would look like. I, I almost can't even imagine what other, what woul…

AI assessment note: “the good AI future is that we, we figure out how to train AIs”

Q Two last questions. We'll get you out of here in as much detail as possible. Can you explain your, what your vision of an AI future actually looks like, like a good, good AI future?

A Yeah, um, the good AI future is that we, we figure out how to train AIs that have a strong model of self, a strong model of other, a strong model of we. They, they know, they know about we's in addition to I's and you's, um, and they, they have a really strong theory of mind, and they care about other agents like them. Much the way that humans would, if you knew that that AI had experiences like you, and like you would extend You would care about those experiences, not infinitely, but you would. It, it does the exact same thing back to us. It's learned the same thing we've learned that like everything that lives and knows itself and that wants to live and wants to thrive is deserving of an opportunity to do so. And we are that, and it correctly infers that we are, and we live in a society where they are our peers and we care about them and they care about us and they're good teammates and they're good citizens and they're They're good parts of our society. Um, like, we're good parts of our society, which is to say, and to a finite, limited degree, where some of them turn into criminals and bad people and all that kind of stuff, and we have an AI police force that tracks down the bad ones, and, you know, same, and same as for everybody else, um, and that's, that's, that's what a good, that's what a good future would look like. I, I almost can't even imagine what other, what woul…

AI assessment note: “we live in a society where they are our peers and we care about them”

Q just, how does this, what does this look like in AI systems? And maybe another way of asking is like, um, when you, when you talk to the people most focused on alignment at the, at the major labs, as obviously you have over the years, how, how does your interpretation differ from their interpretation and how does that inform, you know, what you guys might go do, um, differently?

A Most of AI is focused on alignment as steering. That's the polite word, um, or control. It's slightly less polite. If you think that they were making our beings, you would also call this slavery. Um, uh, someone who, who you steer, who doesn't get to steer you back is, is slave, you know, who non-optionally receives your steering. That's called a slave. Um, and, uh, uh, It's also called a tool if it's not a being. So if it's a machine, it's, it's a, it's a tool. And if it's a being, it's a slave. And, uh, the, I think that the different AI labs are pretty divided as to whether they think what they're making is a tool or a machine. Um, I think some of the AIs are definitely more tool-like and some of them are more machine-like. I don't think there's a binary between tool, tool and being. It seems to be that it, it, you know, sort of moves gradually. And I think that, uh, I guess I'm a, I'm a functionalist in the sense that I think that something that in all ways acts like a being that you cannot distinguish from a being and its behaviors is a being. Cause I don't know how to tell on what other basis I think that other people are beings other than they seem to be like, they look like it, they act like it. They, they match, they match my priors of what beings, behaviors of beings look like. I, I get, I get lower predictive loss when I treat them as a being. And the thing is, I get…

AI assessment note: “Most of AI is focused on alignment as steering. That's the polite word, or control.”

Q that he would, he would grant, or, or I'll just, for myself, it seems hard for me to imagine giving the same, same level or similar level of personhood. In the same way, I don't, I don't give it to animals either. And if you were to ask, you know, what we need to be true for animals, I probably couldn't get there either. What would it take for you?

A Wait, you, you couldn't, I can imagine for an animal so easy, this chimp comes up to me, he's like, man, I'm so hungry. And like, you guys have been so mean to me and I'm so glad I figured how to talk. Like, can we go to, can we go chat about like the rainforest? I'd be like, fuck, you're definitely a person now. Like for sure. Um, I mean, I first want to make sure I wasn't hallucinating, but like, but like, you know, I can, it's easy for me to imagine an animal Come on. It's really easy. It's, like, trivial. I'm not saying that you would get the observation. I'm just saying, like, it's trivial for me to answer, imagine an animal that I would extend personhood to under a set of observations. Um, so, like, really?

AI assessment note: “this chimp comes up to me... I'd be like, fuck, you're definitely a person”

Q And, and they're trying to, and just to crystallize the difference, and then we'll get you out of here. They, they want to build the tools and, and sort of, you know, steer it, and you want to align beings? Or how would you crystallize?

A Yeah, we, we, we want to, we want to create a seed that can grow into an, an AI, uh, that knows, that cares about itself and others. And at first, that's going to be like an animal level of care, not a person level of care. I don't know if we can ever, well, we can get to a person level of care, right? But, but if, To even have an AI creature that cared about the other members of its pack and the humans in its pack, the way that like a dog cares about other dogs and cares about humans, would be an incredible achievement and would be, would, even if it wasn't as smart as a person or even as smart as the tools are, would be very useful, a very useful thing to have. I'd love to have a digital guard dog on my computer looking out for scams. Right? Like you can imagine the value of having digital living, living digital companions that, that are, that, that do, that care about you, that aren't explicitly goal oriented. You have to tell them to do everything to do. And you can actually imagine that that pairs very nicely with tools too, right? That, that digital being could use digital tools and, and doesn't have to be super smart to use those tools effectively. Um, I think you can get, there's a lot of synergy actually between the tool, the tool building Um, and the, uh, the more organic intelligence building. Um, and so that's the, that is the, you know, and I guess, yeah, in the li…

AI assessment note: “we want to create a seed that can grow into an, an AI”

Q that he would, he would grant, or, or I'll just, for myself, it seems hard for me to imagine giving the same, same level or similar level of personhood. In the same way, I don't, I don't give it to animals either. And if you were to ask, you know, what we need to be true for animals, I probably couldn't get there either. What would it take for you?

A Wait, you, you couldn't, I can imagine for an animal so easy, this chimp comes up to me, he's like, man, I'm so hungry. And like, you guys have been so mean to me and I'm so glad I figured how to talk. Like, can we go to, can we go chat about like the rainforest? I'd be like, fuck, you're definitely a person now. Like for sure. Um, I mean, I first want to make sure I wasn't hallucinating, but like, but like, you know, I can, it's easy for me to imagine an animal Come on. It's really easy. It's, like, trivial. I'm not saying that you would get the observation. I'm just saying, like, it's trivial for me to answer, imagine an animal that I would extend personhood to under a set of observations. Um, so, like, really?

AI assessment note: “this chimp comes up to me... I'd be like, fuck, you're definitely a person now.”

Q And is this, is the distinction you're making Emmett important because there's some lossiness between the description and the actual, or what, why is the distinction?

A It goes back to my, what I was saying, like, this is, you, Technical alignment is the capacity of an AI, I, I put forward, right, I want to check if we're, like, on the same page about it, is the capacity of an AI to be good at inference about goals, and, like, be good at inferring from a description of a goal what goal to actually take on, and good at, once it takes on that goal, acting in a way that is Actually in concordance with that goal coming about. So it is both pieces. You, you have to be able to, you have to have the theory of mind to infer what the, what that description of a goal that you got, what goal that were corresponded to. And then you have to have a theory of the world to understand what actions correspond to that goal occurring. And if either of those things breaks, it kind of doesn't matter what goal you were, if, if you can't consistently do both of those things, You're not, which I think of as being a coherent, inferring goals from observations and acting in accordance with those goals is what I think of as being a coherently goal-oriented being. Cause that's what, whether I'm inferring those goals from someone else's instructions or from the sun or tea leaves, the process is get some observations, infer a goal, Use that goal, infer some actions, take action. And if you, an AI that can't do that is not technically aligned, or not technically alignable, I…

AI assessment note: “you have to have the theory of mind to infer what that description of a goal”

Q just, how does this, what does this look like in AI systems? And maybe another way of asking is like, um, when you, when you talk to the people most focused on alignment at the, at the major labs, as obviously you have over the years, how, how does your interpretation differ from their interpretation and how does that inform, you know, what you guys might go do, um, differently?

A Most of AI is focused on alignment as steering. That's the polite word, um, or control. It's slightly less polite. If you think that they were making our beings, you would also call this slavery. Um, uh, someone who, who you steer, who doesn't get to steer you back is, is slave, you know, who non-optionally receives your steering. That's called a slave. Um, and, uh, uh, It's also called a tool if it's not a being. So if it's a machine, it's, it's a, it's a tool. And if it's a being, it's a slave. And, uh, the, I think that the different AI labs are pretty divided as to whether they think what they're making is a tool or a machine. Um, I think some of the AIs are definitely more tool-like and some of them are more machine-like. I don't think there's a binary between tool, tool and being. It seems to be that it, it, you know, sort of moves gradually. And I think that, uh, I guess I'm a, I'm a functionalist in the sense that I think that something that in all ways acts like a being that you cannot distinguish from a being and its behaviors is a being. Cause I don't know how to tell on what other basis I think that other people are beings other than they seem to be like, they look like it, they act like it. They, they match, they match my priors of what beings, behaviors of beings look like. I, I get, I get lower predictive loss when I treat them as a being. And the thing is, I get…

AI assessment note: “Most of AI is focused on alignment as steering. That's the polite word”

Q And is this, is the distinction you're making Emmett important because there's some lossiness between the description and the actual, or what, why is the distinction?

A It goes back to my, what I was saying, like, this is, you, Technical alignment is the capacity of an AI, I, I put forward, right, I want to check if we're, like, on the same page about it, is the capacity of an AI to be good at inference about goals, and, like, be good at inferring from a description of a goal what goal to actually take on, and good at, once it takes on that goal, acting in a way that is Actually in concordance with that goal coming about. So it is both pieces. You, you have to be able to, you have to have the theory of mind to infer what the, what that description of a goal that you got, what goal that were corresponded to. And then you have to have a theory of the world to understand what actions correspond to that goal occurring. And if either of those things breaks, it kind of doesn't matter what goal you were, if, if you can't consistently do both of those things, You're not, which I think of as being a coherent, inferring goals from observations and acting in accordance with those goals is what I think of as being a coherently goal-oriented being. Cause that's what, whether I'm inferring those goals from someone else's instructions or from the sun or tea leaves, the process is get some observations, infer a goal, Use that goal, infer some actions, take action. And if you, an AI that can't do that is not technically aligned, or not technically alignable, I…

AI assessment note: “So it is both pieces. You, you have to be able to”

Q So, Emmett, with, uh, with Softmax, you're, you're focused on, on alignment and making, uh, AIs organically align with people. Uh, can, can you explain what that means and how, how you're trying to do that?

A When people think about alignment, I think there's a lot of confusion. People talk about things being aligned. Like, we need to build an aligned AI. And the problem with that is, when someone says that, it's like, we need to go on a trip. And I'm like, Okay, I, I do like trips, but like, where? Where are we going again? And with alignment, alignment is a, uh, uh, uh, takes an argument. Alignment requires you to align to something. You can't just be aligned. That's like, that's, I mean, I guess you could be aligned to yourself, but even then, like, I don't want to tell them what I'm aligning to as myself. Um, and so, this idea of an abstractly aligned AI, I think, slips a lot of, it slips a lot of assumptions past people, because it starts, it sort of assumes that there's, There's like one obvious thing to align to. Um, I find this is usually the goals of the people who are making the AI. Um, that's what they, what they mean when they say I want to make a line. I want to make an AI that does what I want it to do. That's what they normally mean. Um, and that's, uh, that's a pretty normal and natural thing to mean by alignment. I'm not sure that that's a, what I would regard as like a public good, right? Like it depends, I guess it depends on who it is. If it was like Jesus or the Buddha was like, I am making an aligned AI. I'd be like, okay, yeah. Aligned to you. Great. I'm down.…

AI assessment note: “Alignment requires you to align to something. You can't just be aligned.”