The Exchanges

Q tools. So it's like sit and clawed code and, uh, your point about being more ambitious than you naturally Uh, feel like being, because maybe it'll actually accomplish the thing. This tip of trying it three times, so the idea there is, it may not get it right the first time, so is the tip there, ask it in different ways, or is it just like, try harder, try again?

A Yeah, I mean, you can just literally ask the exact same question. These things are stochastic, and sometimes they'll figure it out, and sometimes they won't. Like in, in every one of these model cards, it always shows like pass at one versus pass at n, and that's exactly the thing where they, they try the exact same prompt. Sometimes it gets it, sometimes it doesn't. Um, so that's, uh, that's the dumbest advice. But yeah, I think if you want to be a little bit smarter about it, there's, there can be Gains there of, of saying like, here's what you already tried and it didn't work. So don't try that. Try something different. Um, that can also help.

AI assessment note: “you can just literally ask the exact same question. These things are stochastic”

Q know you spent a lot of your time on safety. I know that's, as you, as you just alluded to, this is a core part of how you think about AI. Um, and I want to talk about why that is, but first of all, just how do you, how do you do, how do you think about this tension between focusing on safety while also not falling way behind?

A Yeah. So initially we thought that it would be, uh, sort of one or the other, but I think since then we've realized that it's actually kind of convex in the sense that like working on one helps us with the other thing. So initially, uh, like when Opus three came out and we, we were finally at the frontier of model capabilities. One of the things that people really loved about it was the character and the personality. And that was directly a result of our alignment research. Um, Amanda Askell did a ton of work on this, and as well as many others, uh, who tried to figure out, like, what does it mean for an agent to be helpful, honest, and heartless? And what does it mean to be in difficult conversations and show up effectively? How do you do a refusal that doesn't shut the person down, but makes them feel like they understand why the agent said, I can't help you with that, uh, Maybe you should talk to a medical professional, or maybe you should, uh, like consider not trying to build bioweapons or something like that. So yeah, I guess that's, that's part of it. And then another piece that's come out is constitutional AI, where we have this list of natural language principles that leads the model to, to learn how we think a model should behave. And they've been taken from things like the UN declaration of human rights and Apple's privacy policy, Uh, Terms of Service and, uh, a whol…

AI assessment note: “we've realized that it's actually kind of convex in the sense that like working on one helps”

Q how Dario, every, every prediction Dario's had about the progress AI is going to have is just spot on year after year. And he's, you know, predicting 2027, 28 AGI, something like that. So these things start to get real. How do you, I guess, what's your response to folks that are just like, ah, these guys are just trying to scare us all just to, you know, get attention?

A I mean, I think part of why we publish these things is we want other labs to be aware of, of the risks. And yes, there, there could be a narrative of we're doing it for attention, but honestly, like from a attention grabbing thing, I think there is a lot of other stuff we could be doing that, uh, Would be more attention grabbing if we didn't actually care about safety. Um, like a tiny example of this is we published a computer using agent reference implementation in our API only, because when we built a prototype of a consumer application for this, we couldn't figure out how to meet the safety bar that we felt was needed for, for people to trust it and for it not to do bad things. And there are definitely safe ways to use the API version that we're seeing a lot of companies use for, for, uh, automated software testing, for example, in a safe way. So we could have, like, gone out and hyped that up and said, Oh my god, Claude can use your computer, and like, everybody should do this today. But we were like, it's just not ready, and we're gonna hold it back till it's ready. So, I think from like a hype standpoint, our actions show otherwise. From a, like, Doomer perspective, It's a good question. I think my personal feeling about this is that, uh, things are like overwhelmingly likely to go well, but on the margin, almost nobody is looking at the downside risk and the downside ris…

AI assessment note: “from a attention grabbing thing, I think there is a lot of other stuff we could be doing”

Q Wow. What fulfilling work, uh, for folks that are inspired with this, I imagine you're hiring for folks to help you with this. Maybe just share that in case folks are like, what can I do here?

A Yes. Uh, so I think 80,000 hours is the best guidance on this for a really detailed look into, like, what do we need to make the, the field better? But a common misconception I see is that in order to have impact here, you have to be an AI researcher. I personally actually don't do AI research anymore. I work on product at Anthropic and product engineering, and we build things like cloud code and model context protocol and, uh, a lot of the other stuff that people use every day. And that's really important because without an economic engine for our company to work on, uh, and without being in people's hands all over the world, uh, we won't have the mindshare policy influence and, uh, revenue to fund our future safety research and, and have the kind of influence that we need to have. So if you work on product, if you work in finance, if you work in, uh, food, you know, like people here have to eat. Um, if you're a chef, like we need all kinds of people.

AI assessment note: “if you work on product, if you work in finance... we need all kinds of people.”

Q is, is this whole, uh, Zuck coming after all the top AI researchers, offering them a hundred million dollar signing bonuses, a hundred million dollar comp. He's poaching from all the top AI labs. I imagine that's something you're dealing with. I'm just curious, what are you seeing inside Anthropic, and just, what's your take on the strategy? What do you think, where do you think things go from here?

A Yeah, uh, I mean, I think this is a sign of the times. Like, this, the technology that we're developing is extremely valuable. Um, our company is growing super, super fast. Uh, many of the other companies in the space are growing really fast. And at Anthropic, I think we've been maybe much less affected than many of the other companies in the space because people here are so mission-oriented. And they stay because, you know, they get these offers and then they say, well, of course I'm not going to leave because my best case scenario at Meta is that we make money. And my best case scenario at Anthropic is we like affect the future of humanity and, um, try to make AI flourish, uh, and, and human flourishing go well. So to me, it's, it's not a hard choice. Other people have different life circumstances and it makes it a much harder decision for them. So for anybody who does get those mega offers and accepts them, I can't say I hold it against them when they accept it, but it's definitely not something that I would want to take myself if it came to me.

AI assessment note: “at Anthropic, I think we've been maybe much less affected”

Q Cool. Uh, so the idea here is this team works with the latest technologies that you guys have built and Explores what is possible. Is that the general idea?

A Yeah. Um, and I guess, uh, I was part of Google's area one 20 and I've read, uh, about like Bell Labs and how to make these innovation teams work. It's really hard to do right. And I wouldn't say that we've done everything right, but I think we've, Done some, like, serious innovation on, on the state of the art from company design, and Raph has been right at the center of that. Uh, when I was first fitting up the team, the first thing I did was hire a great manager, and that was Raph. Um, and so he's definitely been crucial in, in building the team and, and helping it operate well. And we defined some operating models, like the journey of an idea from prototype to product, and how should graduation of products and projects work? How do teams Uh, do sprint models that are effective and, uh, and make sure that they're working on the right ambition level of thing. Um, so that's been really exciting. I guess, uh, concretely, we think about skating to where the puck is going, and what that looks like is really understand the exponential. Um, there's this great, uh, study that Meter has done that, uh, Beth Barnes is the CEO of that organization, and Uh, shows, like, how long a time horizon of software engineering task can be done. And just really internalizing that of, like, okay, don't build for today. Build for six months from now. Build for a year from now. And the things that are…

AI assessment note: “Yeah. Um, and I guess, uh, I was part of Google's area one 20”