AI Safety

“So if you think it's so totalizing, then you don't make the bunker. And so saying, hey, you don't have a bunker is not proof that the person doesn't believe what they're saying.”

“It's so easy to condense a PDOOM down to a single number, but I think people like stories. They want something that actually illustrates the chain of events that lead to the bad scenario, as opposed to these vague things where it's like, I have a lot of author…”

“The way you do things is to put things out into the world, get feedback from real customers, see where they break, see where they don't break. That is the way you make a good product. That is also the way you make a safe product.”

“From this point onward, probably AI safety is relevant not only for deployment but also during training and evaluation. Like these models might be quite powerful even before they are sort of released to the general public. So that's not the only point at which…”

“One dystopia that I'm particularly nervous about 10 years from now is we overreact to AI safety, and so we say, look, everyone, you're gonna get a cure for cancer, you're gonna have material abundance, but you will have no freedom, you will have no agency, it …”

“Safety is not something you can procrastinate on because capability curves tend to be sigmoid shaped and it can look like everything's fine and then all of a sudden there's something that you didn't Anticipate the model being able to do.”

“And when it comes, we've no idea how to be safe.”

“I think actually one of the big trends we're seeing is a lot of safety is moving from the model level to the ecosystem level and talking about, you know, what's not one model capable of, but what's AI broadly capable of.”

“What's so scary when I was at this conference, which is only about 35 or 40 people, with one modeler from the four biggest model companies, and when I was able to ask them pointedly, How do you think that AI safety gets resolved? The, pretty much the consens…”

“The first part of the safety of a product is that it performed as advertised. The first part of safety is performance.”

“Larry Page was not you know, he was not taking AI's safety seriously.”

“At the time, a lot of the AI safety discussion was kind of theoretical, like the models weren't actually that good. They weren't really posing these dangers, so it was a lot more like philosophical.”

“I think in this field, the alignment of AI systems and the capability of AI systems is intertwined in this way that always ends up being kind of more tied and more intertwined than we think. Actually what this made me realize is that It's very hard to work on …”

“So I think a lot of people think AI safety is overblown, but I think they ignore the paperclip maximizer problem where you have AI models that are accidentally trained towards the wrong objectives, even though this is a big problem that all the models face tod…”

“If you look at, like, who in the world is actually working on safety problems, it's a pretty small set of people even now. I mean, the industry is blowing up, as I mentioned, like, three hundred billion a year CapEx today, and Then I would say like maybe less …”

“So initially we thought that it would be sort of one or the other, but I think since then we've realized that it's actually kind of convex in the sense that like working on one helps us with the other thing.”

“And I think honestly, through our safety research, we have a big opportunity to do things that no other company can safely do. So for example, with computer use, I think that's going to be our huge opportunity, basically like to make it possible for an agent t…”

“Safety is not a function of technology. It's a function of how we apply it. So like electric motor, you know, You can't, the maker of electric motor can't guarantee that no one will ever use it for an unsafe downstream task. Like using electric, electric motor…”

“Bad actors could use it to, you know, more quickly design, you know viruses and stuff like that. That's still speculative, right? So anytime we are taking the focus away, it's like, has that happened? Right? This is still people writing science fiction, fan fi…”

“My PDOOM score is one percent. Like, I'm not totally, you know, unworried, right?”

“Since it'd be such a big deal we'd need to make sure that we can think about it properly channel it in a productive direction, and take care of some sort of tail risks which will, are generally systematically under addressed.”

“So I view the distinction between alignment and safety as alignment as being a sort of subset of safety. Obviously you want the value systems of the AIs to be in keeping with or compatible with say the US public for USAIs or for you as an individual, but that …”

“Safety isn't, as I've been I'm trying to reinforce not really that much of a technical problem. This is more of a complex geopolitical problem with technical aspects.”

“It makes the case for these AI doomers that are like, guys, we need to stop fighting among ourselves. And it needs to be more of like the humans versus AI. And like, we need to take this seriously. And that was more convincing than any like essay that I've see…”

“I think this is likely what we're doing with AI and AI safety and stuff like that, that we could foresee in 1020 years AGI, where there's actual safety issues, but we are so far from that today. That's, I think, actually getting close to the peak of inflated e…”

“The first is that I think the vast majority of AI safety should not be about these topics. The vast majority should be about quite practical concerns we have on making systems safer, like the kind that I've talked with you about so far.”

“I want to develop and improve safety of these tools Because I want to use them. It's not that we have some moral imperative that we have to develop these tools. I mean, maybe there are, or we have to develop AI and AGI. Maybe that's true. But that's not what m…”

“I think there are certain things like, for example, AI safety, where we very much need to work as a world to help set standards and help better the future of everyone here. Because yes, certain things can be done by countries. Capabilities can maybe advance mo…”

“China is more paranoid about AI safety than probably any other government.”

“We've talked a lot this year about safety, for example, about how to make sure that we have models that are performing well, but also are aligning with the values of peoples and the needs of people which I consider sort of a much more grounded problem that we …”

“And the way you make this healthy competition is not By trusting a couple of companies to do their own safety, it's rather for, it's rather the way you do it is to ask application makers to comply with some rules.”

“If you talk to people who know Ilya, he's like, I'm not EA, I'm AI safety.”

“The larger models are much more easily trainable to say when someone asks for, I'd like to break into the following computer, right? Help me do it. It goes, well, I'm sorry, I can't do that. Today's models, you go, well, my grandmother used to put me to sleep …”

“The thing that I will say is we fiercely compete with a whole bunch of these folks. But, like, one of the things that I don't do is, like, look at any of those companies that you just named and, like, worry that they're going to do something that like, is, lik…”

“And indeed the whole problem I had with the whole concept of AI safety and AI ethics was like, this stuff is all fundamentally open source. So aren't five companies that can do this. Anybody will be able to do this. So you're not going to be able to sit and wr…”