The Ledger, every show

“I think that later on, it becomes so destabilizing that China just says, we're going to do something preemptive, like do a cyber attack on your data center. And the U.S. Might do that to China.”

“I think that form of competition would be very dangerous, and because there's a risk of loss of control, and because it might incentivize states to engage in preventive sabotage or preemptive sabotage to disable these sorts of projects.”

“If the US were pulling ahead both Russia and China may have a substantial interest in saying, hey, cut this out. Pulling ahead to develop super intelligence, which could give it a huge advantage and an ability to crush crush them. They'd say, you don't get to …”

“If you turn the pain dial all the way up for China in export controls and if AI chips are the currency of economic power in the future, then this increases the probability that they want to invade Taiwan.”

“If you do it voluntarily, you just make yourself less powerful and you let the worst actors get ahead of you. You could say, well, we'll try and try to sign a treaty. We will not assume that the treaty will be followed. Like that would be very imprudent. You w…”

“Safety isn't, as I've been I'm trying to reinforce not really that much of a technical problem. This is more of a complex geopolitical problem with technical aspects.”

“Even so, I still think if you really tighten the export controls, you made it so that China can't get any of those chips at all, and this is your, one of your biggest priorities, they're just going to steal the weights anyway.”

“If you could have one AI do that, then you could make, you know, a 100,000 copies of these and have them perform research simultaneously. So that could lead to some very substantial acceleration in the rate of development. You might get a decade's worth of AI …”

“Meanwhile, some automated AI research and development loop that goes extremely quickly with very little human oversight, that seems hard to de-risk and get the risks to be at a negligible level, just because it's so fast and so ah, and there's so little human …”

“So, I don't think AI poses a risk of extinction like today, ok? I don't think that they're powerful enough to do that. They, because they can't make PowerPoints yet, right? They don't have Agential skills. They can't accomplish tasks that require many hours to…”

“In the shorter term when AIs get more agential, I'd be concerned about AIs causing cyber attacks on critical infrastructure, possibly by as directed by a rogue actor. There'd also be the risk of AIs facilitating the development of bioweapons in particular pand…”

“Loss of control risks, which I think primarily stem from people, an AI company trying to automate all of AI research and development, and they can't have humans check in on that process because that would slow them down too much. If you have a human do a week …”

“We are finding that with the most recent reasoning models quite unlike the models from two years ago, like the initial GPT-IV, the most recent reasoning models are getting around 90th percentile compared to these expert level virologists in their area of exper…”

“Them doing brainstorming to come up with ways to make viruses more dangerous, I think that's a capability that they've had for over a year, the brainstorming part, but the implementation part seems to be fairly different.”

“So, I think in bio, actually, the I would not be surprised if in a few months there's a consensus that there expert level in many relevant ways in that we need to be doing something about that.”

“When there's very high performance on, on that benchmark that would be suggestive of something that has, say, in the ballpark of superhuman mathematician capabilities. And so I think that would revolutionize the, academy quite substantially. Because all the th…”

“That sort of paradigm does seem like it's running out of steam. It has held for many, many orders of magnitude but the returns on doing that are lower.”

“That is separate from the new reasoning paradigm that has emerged in the past year which is where you train models to on math and coding types of questions with reinforcement learning, and that has a very steep slope, and I don't see any signs of that slowing …”

“So in, in bio, I think that is offense dominant. If somebody creates a virus, there's not necessarily a cure that it will immediately find for it. If it would help a rogue actor make A somewhat compelling virus. Now that could be enough to cause many millions …”

“Where in the context of critical infrastructure there the software is not updated rapidly. So even if you identify various vulnerabilities, there will not necessarily be a patch, because the system needs to always be on, or there are interoperability constrain…”

“Over, you know, 90% of the intellectual energies that they're going to spend is actually, how can we afford the 10 X larger supercomputer? And, ah, that means being very competitive, speeding this up and making safety be some priority, but not necessarily a su…”

“An individual company pausing their development while others race ahead doesn't make game theoretic sense.”

“I think that overall in terms of cultural influence and people being more disagreeable and doing less self-censoring has been has been successful. I think that was the main objective of it. And so I think I think that X had a large role to play there. So I don…”

“When you get to a different paradigm, like automated AI R&D, the slope might be extremely high, such that if the competitor starts to do automated AI R&D a year later, they may never catch up just because you're so far ahead and your gains are compounding on y…”

“This is why I think it might take a while for superintelligence to be developed, because there'll be deterrence around it later on.”

“So we have a paper out last week, we're just measuring the extent to which they're deceptive. And in the scenarios we have, like all the models were in these sorts of scenarios under, you know, slight pressure to lie, not being told to lie, but just some sligh…”

“I think voluntary commitments from AI companies are also a distraction because the companies will, you should expect most of them by default to just break those sorts of commitments if they end up going up against economic competitiveness.”

“If for instance they have these cyber capabilities later on yeah, I think that, or I think that would be a potential place to be drawing the line on on open weight releases personally. In particular the ones that could cause damage to critical infrastructure.”

“China can have AIs that are totally aligned with them. The U S can have AIs that are totally aligned with them. You still are going to have a strategic competition between the two. This is going to they're going to need to integrate it in their militaries. The…”

“The AIs are getting very good at STEM PhD level types of topics, and that includes virology. So I think that they are sort of rounding the corner on being able to provide expert level capabilities in terms of their knowledge of the literature, Or even helping …”

“I mean, we have, you know, some places, you know, 30% plus of the employees at these top AI companies are like Chinese nationals.”

“All they need to do is do a zero day on Slack. And then they can know what DeepMind is up to in very high fidelity and OpenAI and XAI and others.”

“I think those risks would potentially grow in time. I don't think they're as substantial now compared to just the malicious use sorts of risks.”

“So I view the distinction between alignment and safety as alignment as being a sort of subset of safety. Obviously you want the value systems of the AIs to be in keeping with or compatible with say the US public for USAIs or for you as an individual, but that …”

“For cyber, I don't think AIs are that relevant for Being able to pull off a devastating cyber attack on the grid by a malicious actor currently.”

“So you can restrict their intent, which is what deterrence does, but I don't think you can reliably or robustly restrict their capabilities.”

“They're in the ballpark of, like, 10 to 20% overall. They're the very best models.”

“There has been a fair amount of generalization from training on coding and mathematics to other sorts of domains like law, for instance.”

“My appointment at XAI, I get a dollar a year. At Scale, I've at Scale AI, I've increased my salary exponentially to where I get 12 dollars a year, a dollar per month from Scale.”

“No, like, super weapon type stuff, but more conventional type of warfare, like drones and things like that, I expect that they'll continue to race and Probably not, maybe not even coordinate on anything like that, but that's just how things would go.”

“When performance is near the ceiling, I think that'd basically be an indication that, like, you have something like a superhuman mathematician or a superhuman STEM scientist for, in many ways, for when they're, when closed-ended questions are very useful, such…”

“This could possibly change overnight, but it's still near the floor. I think they're still extremely defective as agents.”

“Since it'd be such a big deal we'd need to make sure that we can think about it properly channel it in a productive direction, and take care of some sort of tail risks which will, are generally systematically under addressed.”