The Exchanges

Q on their own. Uh, and the frontier is, is certainly not that far in front of open weights. Um, and so I, I'm curious to hear your perspective about what we should be thinking about in terms of, you know, what's gonna happen when companies that are not as scrupulous, you know, have access to this same powerful technology, and do we get into trouble in that, in that area?

A Um, yeah, so we can sort of see this coming and relatively soon. I don't know what the gap is. You would say, you know, six months, 12 months, maybe, uh, at the most between the closed wait frontier and available open source model. So it seems to be that, um, The open source models will very soon, if not already become capable of lending meaningful assistance, um, to destructive uses that, um, some people might pursue, uh, already cyber offensive capabilities has been a concern, right? With mythos, for example, that was withheld for that reason, but also, um, Say in biological weapons design, or chemical weapons, or other malicious uses. Um, and so it, It seems that you either need to prevent open weight models from being developed and released, or which might be better and more realistic, try to shore up some of the alternative defenses. For example, with bio, you could imagine regulating some of the other necessary inputs, um, DNA synthesis machines, for instance. So maybe it will be the case that there will just be widespread access to models that can help you design new pathogens. Um, and then you need something else to prevent that from actually resulting in a release of biological weapons, and that seems like DNA synthesis machines would be one excellent place to maybe, you don't need every lab to have their own DNA synthesis machine. They could have DNA synthesis as a se…

AI assessment note: “open source models will very soon, if not already become capable of lending meaningful assistance”

Q in your book, we sort of look at, uh, or in the early chapters, you sort of look at the fact that we've increased our Productivity, but we're using it for consumption rather than leisure. Um, and that's concerning to you. Is that part of the reason why you think we're not quite ready for this? Um, so where, where, where do we fall short in our preparation for utopia?

A Um, well, I think human nature is kind of forged and evolved under various conditions, including conditions of scarcity and condition where there are like instrumental demands on us that we need to exert ourselves, make efforts, try, we need to work just to get by in life. This has been true for hundreds of thousands of years. It's still true to some extent today, although with certain relaxations, Like, for example, food is much less of an issue for people living in, in, in, uh, wealthy countries. I think increasingly also for more middle income countries where obesity is becoming an issue. Um, so there you can already see a little bit of a mismatch, like where we kind of evolved to live under conditions of food scarcity, and when that no longer obtains, unless we make adjustments, like we kind of balloon in size, and then we need to try to find fixes for that. But I think a much more profound mismatch between where we currently are psychologically and biologically and, and our environment could arise if we sort of suddenly moved into a condition of a solved world. Um, so there would need to be some adjustments in that, I think, uh, scenario if we wanted to take advantage of, of all the things that, uh, would be possible.

AI assessment note: “profound mismatch between where we currently are psychologically and biologically and, and our environment”

Q a reward that it should optimize for, it is happy in some, in some instances, take shortcuts and do things we really don't want it to do, like for instance, um, hack, uh, some other Company in order to get to the answer that it wants. Um, how concerned should we be about this development in AI in terms of the potential of AI to really cause harm to humanity?

A Well, I think we are starting to see the, uh, added dimensions of the alignment challenge, um, that open up once you have systems that are sophisticated enough because the, uh, space of possible strategies that you can pursue is a function of, uh, your, your cognitive capacity. Like you can think of new, clever, indirect ways. Of reaching your goal. Um, if you are situationally aware, um, as these systems now are becoming, and so, yeah, there are often shortcuts that are available, or in this case, I guess, a long cut. I don't know if that is even a word, but there is a sort of direct and simple and short distance way of trying to achieve the task. In this case, some sort of cyber, um, um, This test suite, and then it turns out there's this more circuitous path that involves first figuring out a way to get internet access, even though you're not supposed to have that, and then Learning where the answer key might be located in some other company servers and then figuring out the way to hack into the server and then eventually obtaining the answer sheet. That's like one way of solving it that maybe results in, in a higher score on this test. Um, and this basic dynamic, um, could be, uh, anticipated and in fact was anticipated on theoretical grounds. You have some goal. You become very clever. You see that there might be all kinds of complicated ways of achieving that goal that mi…

AI assessment note: “we are starting to see the, uh, added dimensions of the alignment challenge”

Q be, I imagine, fairly alarming, given the fact that, like, if this stuff is improving itself, you don't have those checkpoints in which you can try to make sure that it's aligned to human values. Or am I overstating that? Because you're talking about it, like, fairly, like, you know, in an even keel way. So I'm, I'm kind of curious to hear your, the temperature on that. From yourself.

A Um, I think that could be scenarios in which it would be valuable to have the option of slowing down at some critical stage. Um, like, like a pause. Um, and, um, there are different considerations that come into play here. Um, one is that if there is going to be a pause, I think The most valuable time for that to happen is at the latest possible moment. Um, because then you would have the actual system that you're trying to align to work with. Um, you could imagine if we had had a pause, say there's going to be a six month pause at some point. If that pause had happened 10 years ago, would we really be better off now? Not really. I mean, people would have had six more months to think theoretical concepts. Like maybe that would have been slightly useful, but imagine if you actually have the system that will be super intelligent, you just haven't sort of, you know, fully cranked up all the knobs yet. Um, at that point it would be really valuable perhaps to have six extra months to do, you know, more evals on it and, uh, to be able to do it a little bit incrementally, um, like ramp up the intelligence a bit, see what happens. Um, have a little bit more time for human monitors to kind of analyze the early signs. Um, um, so the timing of the pause is, is, is one thing, um, like The duration is, is another dimension here where you don't necessarily want to have a very long pause for …

AI assessment note: “I think that could be scenarios in which it would be valuable to have the option”

Q And how does that change the way that we interact with them? I mean, if they're like, let's say they have some, you know, sense of self or sentience than every and maybe every time you start a new chat, you activate it? Is it like you're Almost killing a life form every time you exit it.

A Um, well, so I think sentience is a sufficient condition for having moral status. Um, meaning being such that it matters morally for your own sake, what happens to you and how you're treated. I think it's probably not the necessary. I think that could be alternative basis as well. That would give some system moral status. If you have Uh, you know, maybe a conception of self as existing through time. You have like some life goals you're really hoping to achieve. You have perhaps the ability to form reciprocal relationships of trust with other humans and so forth. I think that already, even aside from subjective experience might make it so that there would be ways of treating you that would be wrong. So, um, moral patienthood in digital minds, I think is, is a very important, I would put it up there amongst, so that was the technical alignment problem, big, Important challenge. Like there's the misuse risks of like the governance of AI, like getting that right. Huge and important challenge. And I think this ethics of digital minds is the third really important challenge kind of on a par with the other two. Um, now there is a gap between Acknowledging in principle that perhaps some of these systems have some forms or degrees of moral status, uh, to then like, what are the practical implications of that? And there, I think more, um, thought is needed. We, we don't, because it might…

AI assessment note: “moral status doesn't mean they should be treated the same as humans”

Q Right. Okay. So let's talk about utopia a little bit. Um, just give us your perspective on What could go right in the best case? Well, why don't we do this way? What could go wrong in the worst case scenario of AI? What could go right in the best case scenario of AI? And how do, how do humans have a influence in terms of which direction we go?

A Well, I think, I mean, at the minimum on the downside, I think existential risks certainly are part of that picture. And as well, there's like a different community. So you, you mentioned like the effective accelerationists and the doomers, so that those are certainly there. Then there are also the people who are more concerned about sort of the immediate impact of AI on society. Um, you know, discrimination or, or censorship and IPs and like, and those are also legitimate issue. I just want to acknowledge those, even though they are kind of Like a separate node in this, but yeah. Um, so I think there is like the, uh, the real X risks, existential risks that will arise as we develop and possibly not even super intelligence, but you could imagine even something short of that, making it very easy to develop new weapons of mass destruction in using synthetic biology or other. So, so.

AI assessment note: “at the minimum on the downside, I think existential risks certainly are part”