The Ledger, every show

“The open source models will very soon, if not already become capable of lending meaningful assistance to destructive uses that some people might pursue already cyber offensive capabilities has been a concern, right? With mythos, for example, that was withheld …”

“Current LLMs, like you're using them as an ordinary person, for the most part, they are helpful, and they try to solve your task that you assign them, or give an answer that is, sometimes they hallucinate, or maybe deceive a little bit, but broadly speaking, t…”

“Well, I think whatever we do there will be both existential risks and individual risks. So it's not as if we have a choice between avoiding risks and confronting risks. So it's looking at these different alternatives and weighing up the risks and benefits. And…”

“I think it's plausible that some AI models have some forms of subjective experience by now. Obviously there's a lot of uncertainty about this, but it does seem That it is efficiently likely that I think we should start to do some things for the sake of these A…”

“Once you start to engage in activism, then There is kind of mimetic pressures to simplify our message and to close ranks and to try to beat the people with opposing views down in the marketplace of ideas. and I think we are seeing some of the beginnings of th…”

“From this point onward, probably AI safety is relevant not only for deployment but also during training and evaluation. Like these models might be quite powerful even before they are sort of released to the general public. So that's not the only point at which…”

“DNA synthesis machines would be one excellent place to maybe, you don't need every lab to have their own DNA synthesis machine. They could have DNA synthesis as a service, and maybe there could be five or six companies worldwide, Or legitimate research labs ca…”

“If you get a kind of weak super intelligence that is For the most part aligned, we might then be able to use that to make a more powerful form of super intelligence that is more reliably aligned.”

“I'd say about the same overall. I mean, there's like some disconcerting signs, but also some positive signs advances in, like, some insights are being gained into how these systems work, and how one can steer them, and so forth. So how to tote that all up, I'd…”

“Turns out there is like some big hobbling that we have unwittingly, like some, something we were doing wrong that just made these systems way less efficient than they could be. And when somebody figures out how to remove that, like maybe the current Compute is…”

“I think we have to take seriously both that we might be relatively close, potentially very close and that Once we get there, you really get a very fast takeoff.”

“The most valuable time for that to happen is at the latest possible moment. Because then you would have the actual system that you're trying to align to work with.”

“So if the pause only applies to the most responsible actors, for example, then a long pause would remove the initiative from the most responsible AI developers and shift it over to the Less responsible AI developers who decide not to abide by the past”

“Another is that you might with a longer pause start to build up a lot of hardware overhang. It's a like, if we keep building out bigger data centers and chips are getting better than a long pause would result in a situation where you now have such a massive am…”

“At the point where it is as good as an average human in everything, it might already be super intelligent in some key relevant domains for AI research.”

“And these studies have been made and any particular, you can go in with a kind of steering vector that suppresses, say deception and role playing. And it turns out when you do that, they become more likely to report that they are conscious and have subjective …”

“And so there was a recent paper by Anthropic looking at the existence of a kind of global workspace inside these large language models. This is the idea of there being a kind of almost like a stage inside a mind where some small subset of all the information t…”

“Moral patienthood in digital minds, I think is, is a very important, I would put it up there amongst, so that was the technical alignment problem, big, Important challenge. Like there's the misuse risks of like the governance of AI, like getting that right. Hu…”

“It's harder to sometimes see the cost of limiting ourselves in all those ways, because the new innovations that would be unlocked are not yet there. Whereas like the pollution is, is there immediately, right? Or the person, like if a self-driving car runs over…”

“So I think there is like the real X risks, existential risks that will arise as we develop and possibly not even super intelligence, but you could imagine even something short of that, making it very easy to develop new weapons of mass destruction in using syn…”

“Well, I think they are both worth worrying about. I think with the X risks, I mean, maybe a slightly larger on the AI being the kind of agentic part there.”

“We often define our sense of self-worth on, on the idea of being a contributor. Like you're a breadwinner, or you make like a positive difference in the lives of your friends or of society at large. You're like, you bring value to the world. So much of our exi…”

“So all these sort of science fiction-like technologies that maybe we would develop if we had like, you know, 20,000 years for human scientists to work for it, we probably will have a cure for aging and perfect virtual reality and space colonies and all the res…”

“You would say, you know, six months, 12 months, maybe at the most between the closed wait frontier and available open source model.”

“I think for cybersecurity right now we're in a regime where attackers often win but it might be that in the limit if you have sort of An AI trying to find vulnerabilities and also patch vulnerabilities and you keep making the AI stronger, like eventually maybe…”

“And patches are a lot easier to roll out in the digital space. So, so maybe there's like some cyber thing. We figure out what the vulnerabilities we can release the patch. And then in, in principle, like almost immediately around the world, all the relevant sy…”

“It's also conceivable that even when you do get recursive self-improvement, you still might not have An intelligence explosion. It might, there might be diminishing returns at some point. Presumably there are at some point, but it could turn out that that is c…”

“This gives us more sort of surface area to work with. Like you can more easily understand and interact with these systems because they have human double concepts and you can talk with them.”

“Anthropic has given Claude a bail button, a tool that it can invoke if it feels that the conversation is abusive to it, that can choose to terminate that session, which is a nice start.”

“If kind of these, you know, development efforts in, in, in, in hardware and in, in algorithms continue, then it looks very likely that we will succeed in this.”

“I think some of the, Impetus for effective accelerationism comes from that kind of cumulative frustration. And then it's applied to AI in particular.”

“In a solved world, in utopia, the utopians could have, like, extreme levels of motivation and immersion and subjective purpose. That's easy. That's like a check mark. And more broadly, you can go through different plausible human values. And for some of them, …”

“Other things like cures for aging and stuff like that, like we don't have them yet, but there is no, you know, no laws of physics prevent people, you know, from living indefinitely.”

“AI, you know, the goal of AI has all along been not just to automate a few specific things, but to provide the technology that allows us to automate all tasks, right? Like AI hasn't really succeeded until all intellectual labor can be done by machine.”

“The simulation hypothesis expands the space of sort of realistic possibilities and the realist, like the space of realistic futures. You might think if you're living just in a simple materialistic universe, you die. That's the end. Your brain rots. There is no…”

“Well, I think we are starting to see the added dimensions of the alignment challenge that open up once you have systems that are sophisticated enough because the space of possible strategies that you can pursue is a function of your cognitive capacity. Like yo…”

“You're focusing there on the misuse potential that this people might choose to do bad things with AI technology. And that certainly is one big category of risk, right? But that's not primarily a technical challenge. It's more ultimately a governance challenge …”

“There is this phenomenon that like, before it is done, it looks really hard. Once AIs have done it, then we kind of quickly forget how, just how like impressive it was and just take it for granted.”

“So, I mean, I've never described myself As an effective altruism, I think my, ah, some of my ideas have had an influence there, and in particular yeah, try to think more about the sort of the macro strategic aspects of our strivings, as opposed to just the loc…”

“We are designed in such a way that our reward system motivates us to produce more effort. At whatever level, like, no matter how good our situation is, we are designed to always try to want to make it better, and so we only get reward when things improve, rath…”