The Exchanges

Q If, uh, you all could propose a magical adoption tactically of some policy or action to the current administration, what is the first step here? It is the, you know, we will not build a super weapon and we're going to be watching for other people building them too.

A And so I've sort of been alluding to throughout this whole conversation, like what would the companies do? Like not that much. I mean, add some basic anti-terrorism safeguards, but I think this is like pretty technically easy. This is unlike refusal for other things. Refusal robustness for other things is harder. Like if you're trying to get it like crimes and torts, that that's harder because it's, it's a lot messier. It overlaps with typical everyday interaction. I think likewise here, the, the asks for states are not that challenging either. It's just a matter of them doing it. So one would be the CIA has a cell that's doing more espionage of other states' AI programs. So that way they have a better sense of what's going on and aren't caught by surprise. And then secondly, maybe some part of government, like let's say cybercom, which has a lot of cyber offensive capabilities, um, gets some cyber attacks ready to, um, disable, um, other data centers in other countries if they're looking like they're doing something, running a, or creating a destabilizing AI project. That's it for the deterrence for nonproliferation of, of AI chips to rogue actors in particular. I think there'd be, um, some adjustments to export controls. In particular, just knowing where the AI chips are at reliably. We want to know where the AI chips are at for the same reason we want to know where our fissi…

AI assessment note: “one would be the CIA has a cell that's doing more espionage”

Q want to change tax for our last couple of minutes and talk about evals. Um, and it's obviously very related to, uh, safety and understanding where we are in terms of capability. Can you just contextualize where, where you think we are? Uh, you came out with a triggeringly named humanity's last exam eval, and then also Enigma, um, like why are these relevant and where are we in evals?

A Yeah, yeah. So for context, I've been making evaluations to try and understand where we're at in this, um, in AI for, uh, I don't know, about as long as I've been doing AI research. Uh, so previously I've done some, um, data sets like MMLU and the math data set. Before that, before ChatGPT, there's, Things like ImageNet-C and, and other sorts of things. So Humanity's last exam was basically an attempt at getting at what's the, what would be the, um, end of the road for the evaluations and benchmarks that are based on exam-like questions, ones that test some sort of academic type of knowledge. So for this, we asked professors and researchers around the world to submit a really challenging question And then we would add that to the, the data set. So it's a big collection of what professors, for instance, would encounter as challenging problems in their, in their research that have a definitive closed ended objective answer. With that, I think the genre of here's a closed ended answer where it's, you know, multiple choice or a simple short answer. I think that genre will roughly be expired when performance on this data set is, uh, near the ceiling. So, and when performance is near the ceiling, I think that'd basically be an indication that, like, you have something like a superhuman mathematician, um, or a superhuman STEM scientist for, in many ways, for when they're, when closed-…

AI assessment note: “Humanity's last exam was basically an attempt at getting at what's the... end of the road”

Q At the highest level, like, bundle of weights, increasingly capable. Like, why do we care about AI from a national security perspective? Like, what's the most practical way, uh, it matters in geopolitics or gets used as a weapon?

A I, I think that AI isn't that powerful currently in many respects. So in many ways, it's not actually that relevant for national security. Currently, this could well change within a year's time. I think generally I've been focused on the, the trajectory that it's on, as opposed to saying right now it is extremely concerning. That said, there are some, for instance, for cyber, I don't think AIs are that relevant for Being able to pull off a devastating cyber attack on the grid by a malicious actor currently. That said, we should look at cyber and be prepared and think about its strategic implications. There are other capabilities like virology. The AIs are getting very good at STEM PhD level types of topics, and that includes virology. So I think that they are sort of rounding the corner on, um, being able to provide expert level capabilities in terms of, uh, their knowledge of the literature, Or even helping in practical wet lab situations. So I, I do think on the virology aspect, um, they do have already national security implications, but that's only very recently with the reason reasoning models. Uh, but, uh, in many other respects, they're not as relevant. It's more prospective that it could well become the way in which a nation might, um, try and dominate another nation, um, and the, the backbone for not just war, but also just economic security. Uh, the amount of chips th…

AI assessment note: “I do think on the virology aspect, um, they do have already national security implications”

Q What is a way you actually expect that AI gets used as a weapon? Beyond virology and, and security. Yeah.

A I wouldn't expect, um, uh, a bioweapon from a state actor, from a non-state actor, um, that, that would make a lot more sense. The, I, I, I think cyber makes sense from, from state actors and both non-state actors. Uh, then there's drone applications. These could disrupt, um, other things. These could help with other types of weapons research, like help explore exotic EMPs, um, Could help, um, uh, create better types of drones, could substantially help with situational awareness, uh, uh, so that one might know where, you know, all the nuclear submarines are. Um, some advancement in AI might be able to help with that, and that could disrupt, uh, uh, our second strike capabilities, um, and neutralistic destruction. So, uh, those are some geopolitical implications. It could potentially bear on nuclear deterrence, and that's not even a weapon. The example of just heightened situational awareness and being able to The pinpoint where, um, hardened, um, uh, land, uh, nuclear launches are, or where nuclear submarines are, um, is, is just informational, uh, but could nonetheless be extremely disruptive. So, uh, or destabilizing. Outside of that, the, the default conventional AI weapon would be drones, um, which is, um, I don't know if that makes sense that come, or that, that countries would compete on that, and, uh, I think that it would be a mistake if the US weren't, um, trying to do…

AI assessment note: “These could help with other types of weapons research, like help explore exotic EMPs”

Q So, okay. In reaction, uh, you propose along with some, you know, other esteemed authors and friends, Eric Schmidt and Alex Wang, a new deterrence regime, a mutually assured AI malfunction. I think that's the right name. MAME, bit of a scary acronym, and also a nod to mutually assured destruction. Can you explain MAME in plain language?

A Let's think of what happened in nuclear strategy. Basically, a lot of, a lot of states deterred each other from doing a first strike because they could then retaliate. So they had a shared vulnerability. So they're, they were, we're not going to do this really aggressive action of trying to make a bid to wipe you out because That will end up causing us to be damaged. And we have a somewhat similar situation later on, um, when AI is more salient, when it is viewed as pivotal to the future of, of a nation. When people are on the verge of making a super intelligence more, when, when they can say automate, you know, pretty much all AI research, I, I think states would try to deter each other from trying to leverage that to, um, develop it into something like a super weapon that would allow the other countries to be crushed. Or use those AIs to do, um, uh, some really rapid automated AI research and development loop that could, um, have it bootstrap from its current levels to something that's, um, a super intelligent, vastly more capable than, than any other system out there. I think that later on, it becomes so destabilizing that China just says, we're going to do something preemptive, like do a cyber attack on your data center. And the US might do that to China. Um, and Russia get coming out of Ukraine will, you know, Reassess the situation. Get situationally aware. Think, oh, wha…

AI assessment note: “states would try to deter each other from trying to leverage that”

Q Which brings us to one of the biggest questions of our time. How do we navigate the geopolitical implications of superintelligence? Dan Hendricks, the director of the Center for AI Safety, has an answer.

A Let's think of what happened in, in nuclear strategy. Basically, a lot of, a lot of states deterred each other from doing a first strike because they could then retaliate. So they had a shared vulnerability. So they're, they were, we're not going to do this really aggressive action of trying to make a bid to wipe you out because that will end up causing us to be damaged. And we have a somewhat similar situation later on, um, when AI is more salient, when it is viewed as pivotal to The future of, of a nation. When people are on the verge of making a super intelligence more, when, when they can say automate, you know, pretty much all AI research, I, I think states would try to deter each other from trying to leverage that to, um, develop it into something like a super weapon that would allow the other countries to be crushed or use those AIs to do, um, uh, some really rapid automated AI research and development loop that could Um, have it bootstrap from its current levels to something that's, um, a super intelligent, vastly more capable than, than any other system out there. I think that later on, it becomes so destabilizing that China just says, we're going to do something preemptive, like do a cyber attack on your data center. And the U.S. might do that to China. Um, and Russia, get coming out of Ukraine, will, you know, reassess the situation, um, get, get situationally aware,…

AI assessment note: “states would try to deter each other from trying to leverage that”