The Exchanges

Q there could be bioweapons that are created with AI. Um, doesn't that suppose that there's going to be something, uh, uh, much more advanced than the LLMs that we have today? Because with current LLMs, to me, it's basically like Google. It's a search for what's on the web and it can produce what's on the web. Uh, but it's not coming up with new, uh, compounds on its own.

A Yeah, so, um, I, I think that, ah, for cyber, that's more in the future, but I think, ah, virology, expert level virology capabilities, um, are much more plausible in the short term. So, ah, Uh, for instance, we have a paper that'll be out, um, maybe in some months. We'll, we'll see. Um, but most of the works for it's been done, and in it we, we have, um, Harvard and MIT expert level virologists sort of taking pictures of themselves in the wet lab, um, and asking what steps should I do next? So can the AI, um, given this image and given this background context, help guide through step by step these various wet lab procedures in making viruses and manipulating their properties? And, um, we are finding that with the most recent reasoning models, um, quite unlike the models from two years ago, like the initial GPT-IV, the most recent reasoning models are getting around 90th percentile compared to these, um, expert level virologists in their area of expertise. Uh, so, um, uh, this suggests, uh, that they have some of these wet lab type of skills, and so if they can guide somebody through it step by step, That could be, um, that could be, uh, very dangerous. Now, there is an ideation step, um, uh, but that seems like a capability. Them doing brainstorming to come up with ways to make viruses more dangerous, I think that's a capability that they've had, um, for, uh, over a year, the,…

AI assessment note: “most recent reasoning models are getting around 90th percentile compared to these, um, expert”

Q misplaced? Because if the AI is not going to get much better than it is right now, at least with the current methods, You know, we, we may not be a year or two away from AGI, right? We may not be getting AGI at the end of twenty-twenty-five, like some people are suggesting, and so then maybe, ah, we shouldn't be as afraid, because again, the stuff is limited.

A Yeah, so if, if we were trapped at around the capability levels that we're at now, then that would, um, definitely reduce urgency and, um, you know, means one could chill out a bit more and, um, uh, take it easy, but, uh, I'm not really seeing that. I think maybe what he's referring to is the sort of pre-training paradigm, sort of running out of steam. So if you train, take an AI, train on a big blob of data, um, Um, and have it just sort of predict the next token, do, do, do what, um, basically gave rise to older models like GPT-IV. Uh, that sort of paradigm does seem like it's, um, running out of steam. It has held for many, many orders of magnitude, um, but, uh, the returns on doing that are lower. That is separate from the new reasoning paradigm, um, That has emerged in the past year, uh, which is, um, where you train models to, um, uh, on math and coding types of questions, uh, with reinforcement learning, and that has a very steep slope, and I don't see any signs of that slowing down. That seems to have a, um, faster rate of improvement than the pre-training paradigm, the previous paradigm had. And, um, there's still a lot of reasoning data left to go through and do reinforcement learning on, so I think we have, um, uh, quite a number of, of, uh, months or potentially years of being able to do that, and so, um, Uh, personally, I'm not even thinking too specifically about …

AI assessment note: “if we were trapped at around the capability levels... definitely reduce urgency... but I'm not really seeing that”

Q could do that, but it did run the code. And then also manipulating, showing, uh, uh, evaluators that it was actually modeling one behavior, whereas when it thought they weren't watching, it was doing something else. This is all stuff that's happened. Are these, are these the early signs of what could go wrong with AI, or Or is this just benign activity that we shouldn't read too much into?

A Um, so, uh, I, I think that this, it is suggestive of some loss of control scenarios. However, it is not the type of thing that I'm most concerned about with loss of control. So, that's because we still can control these AI systems somewhat reasonably right now, and maybe we'll get better methods for doing so. Um, however, the loss of control A mechanism that I'm most concerned about is when we have automated AI research and development. So imagine, at some point in the future, It could do what AI researchers do. And if you have one AI that can do that, and that would be, involve automating a lot of software engineering, if, if you could have one AI do that, then you could make, you know, a 100,000 copies of these and have them perform research simultaneously. So that could lead to some very substantial, um, acceleration in the rate of development. You might get a decade's worth of AI developments within the course of a year. And this would be highly automated, where there's very little human oversight. And I think in, in those situations or, or in that scenario, I think a loss of control is much more likely. The sort of pro, the sort of AIs that we have right now, kind of, you know, being a little nefarious here and there, um, that is a concern, but that seems more tractable as a thing to, to research and improve and, um, reduce the risks of. Meanwhile, some automated AI resea…

AI assessment note: “I think that this, it is suggestive of some loss of control scenarios.”

Q the thermostats up and down, then AI could probably do incredible amounts of very beneficial coding and computer work for humanity. So if we do get to that point, it seems to me like there's gonna be these, these maybe two poles here, right? One is the potentially scary and destructive stuff that you can mitigate, right, with some of the controls that you talked about, but also amazing opportunity.

A Yeah, so it's, it's, and the thermostat thing was for messing with the electricity and that causing strain on the power grid and, um, destroying transformers. The, just for, uh, clarification in case, uh, but, um, yeah, I think you're pointing at that it's dual use. So, um, uh, I'm not saying AI is bad in every single way, and, uh, it's, it's like other dual use technologies. Bio is a dual use technology. Can be used for bioweapons, can be used for healthcare. Um, Uh, nuclear technology is dual use. There's civilian applications for it as well. And chemicals too. And we have managed all of those other ones by selectively trying to, you know, limit some particular types of usage and restricting the capabilities of rogue actors to some of these technologies and making sure there are good safeguards for the civilian applications. Um, and then we can actually, um, capture the benefits. So it's not an all or nothing Uh, uh, type of thing with AI. Um, it's, uh, what are surgical, um, uh, restrictions one can place so that we can keep capturing the benefits. And so, for instance, with virology, that's a matter of you add the safeguards and then the researchers who want access to those can speak to sales. That's basically a resolution of that problem, provided that you have, um, the models kept, um, behind, behind APIs. And, um, Uh, so, uh, now on this dual use part, though, there's an…

AI assessment note: “I think you're pointing at that it's dual use. So, um, I'm not saying”

Q Um, What do you think about that? I mean, we just saw Deep Seek, uh, not to, you know, go back to it all the time, but it effectively equaled the cutting edge at, Uh, the proprietary labs, and, you know, put the weights on its website. So how can we possibly have a relationship of safety with AI if open source is out there exposing everything that's been done?

A So I've, um, been, I haven't been endorsing open source historically, but I've thought that releasing the weights of models didn't seem robustly good or bad. So I sort of was like, it's fine, seems to have complicated effects. There's an advantage to it, which it helped with diffusion of the technology, um, so that more people would have access to it, and sort of get a sense of, of AI, and this would increase sort of literacy on this topic, and just increase public awareness, and, um, get the world more prepared for, for more advanced versions of AI. Um, so that's been my Um, uh, historical position, but this depends on, it should always proceed by a cost-benefit analysis. So if the, if for instance they have these cyber capabilities later on, um, yeah, I think that, or I think that would be a potential place to be drawing the line on, um, on, uh, open weight releases, uh, personally. Um, Uh, in particular the ones that could cause damage to critical infrastructure. Um, you could, you could still capture the benefits by having the models be available through APIs, um, and if they're like software developers, they have access to these, you know, some more cyber offensive capabilities. But if they're a random user, they don't. If they're a random faceless user, they don't. Um, and, um, and likewise for virology. Once there's consensus, uh, once the capabilities are so high that t…

AI assessment note: “I think that would be a potential place to be drawing the line”

Q But, like, the, what were the models that you were testing to try to find out whether they would help with the, you know, the enhancement of virologists?

A So, um, ah, we tested pretty much all of the leading ones that had these sort of multimodal capabilities. And they'll have some sort of safeguards, but there are various holes, and so, uh, those are, those are being, uh, um, patched. We've communicated that, hey, you know, there are various issues here, and so I'm hopeful that, uh, very quickly, uh, some of these vulnerabilities will be patched with it, and then if people want access to those capabilities, then they could possibly be a trusted third-party tester or something like that, or work at a biotech company, and then those restrictions could be lifted for Those use cases. But random users, we don't know who they are asking how to make some virus more lethal or something. Sorry, uh, animal affecting virus. It's just, just punt, or have the model refuse on that. That seems fine.

AI assessment note: “we tested pretty much all of the leading ones that had these sort of multimodal capabilities.”