The Exchanges

Q the reply to that mentioned, uh, that he had contacted a lawyer. Um, so there's my apology. I still don't love the puzzle, but, um, back to you, Steven, this is, this is a little bit, uh, what would you, what actually, let me just ask you the question without leading the witness. Is it narcissism or is there actually something, something, you know, potentially disconcerting happening behind the scenes?

A I think it's very brave in that by and large, these are people sacrificing very large amounts of money to give the warnings they are. I do wish that they would be more direct. Um, but to put it in context, you know, back in. It seems that OpenAI and Anthropic had secret non-disparagement agreements, which in OpenAI's case, at least, you know, plausibly not permitted by law, the way that they operated this, where To keep your already vested equity, the compensation you had been told was yours. You had to sign away your right to say anything negative about open AI and in fact, sign away your right to tell anyone that you had signed this contract. Um, and this was secret and kept under wraps for years until Daniel Cocotelo, um, who people might know from leading AI, 2027, I think very, very courageously forwent this agreement and forfeited something like 80% of his family's net worth and said, Sorry, I'm, I'm just not waiving my right to criticize open AI. Um, and in the wake of that, you know, there was a bunch of outpouring open AI and anthropic changed the nature of these contracts. And still it's pretty intimidating to speak out against these massively resourced legal operations. Um, you know, not afraid of subpoenaing different people and getting into legal conflict. You want to be really, really careful about what you say. And so in Renox case, I noticed in the footnotes, ri…

AI assessment note: “I think it's very brave in that by and large, these are people sacrificing”

Q Oh, I think that would do good ratings. Okay. Steven, you also talked a little bit about in, in a recent newsletter about how, uh, basically we have very limited regulation on these companies. Uh, and even then they might still not be following. So can you just expand upon that briefly?

A As of 2026, there is finally some amount of law in the United States about how companies are meant to do testing for the catastrophic risks we have talked about. Um, until this point, purely voluntary. This bill is called SB 53. It came into effect in January and it's very, very light touch. It basically says the most major of the AI companies, you need to publish how you are going to test for these risks. You need to do what you said you are going to do, and you can't be misleading about it, but there's no quality standard. You could basically say, we will test for the risks as we deem appropriate and nothing further, and that, that would be fine. But if you say you are going to do this testing, you need to, in fact, follow through on it. Um, and unfortunately, it seems like OpenAI's release of last week, GPT, 5.3 codex, one of the big breakthrough models we've been talking about. As I look over the evidence, It seems like open AI did not abide by the testing that they had committed to in various ways. And so, you know, ultimately this decision now is with the attorney general of California to investigate it, whether to enforce a fine, a pretty small fine, maybe like up to a million dollars compared to open AI, hundreds of billions of dollars in valuation. Um, it just really seems to me like if we care about these risks, letting companies self assess in this framework is reall…

AI assessment note: “This bill is called SB 53... It seems like open AI did not abide”

Q opposed AI mode. But the story does say that there was a group of people within OpenAI who have Within the company, uh, stated their opposition seemingly loudly to the fact that it's going to roll out, uh, this adult mode, and by the way, adult mode is going to be coming out, seems like, in the coming weeks, coming months at most. Stephen, what should we make of this?

A It's, it's hard to weigh in on any one personnel incident. And also, uh, this would not be the first time that OpenAI seems to have done a pretextual firing where they got a person out of the organization who had safety concerns that the company, um, either didn't like or didn't like how they had expressed them. And, and notably those are different, right? Like you can have concerns about how the company is operating and that doesn't mean you have license to Say anything in any forum. Um, but Leopold Ashenbrenner, who wrote this huge essay, situational awareness in the past, um, maintains that open AI said things to him when he was fired that implied it was, it was basically because he had contacted the board about security concerns that opening eyes models were not actually secure. And so I think the way to interpret all of this, right? These, these aren't an apocalypse in the sense of Something super, super substantively scary happening right now, but I think they are early warning signs that people within the companies are raising flags of sorts, and they are not being permitted to speak freely. They are paying consequences for it. And so the question is, as we get to more and more dire issues at some point, hopefully we don't, but we might, you know, will, will we have people who are still sounding the alarm who are willing to have the, the courage of their convictions in t…

AI assessment note: “I think they are early warning signs that people within the companies are raising flags”

Q smarter with the, with the actual model itself, um, you know, doesn't seem right to me. That, to me, felt like the weakest part, and also the part that got most people most alarmed, uh, of the entire essay. So Stephen, to you, what do you think about this recursive self-improvement, uh, uh, argument? And then briefly, just on the, on the, the entirety of the essay itself, your thoughts.

A I think Matt's essay is directionally correct, but a bit early, and there are a few steps that we maybe haven't gotten to yet. Um, I think he is largely correct on the automation of engineering within the AI companies. It's like a little overstated relative to my experience, the experience of people I talk to, but broadly there, there has been a huge shift. The job of an engineer at one of these companies now is much more supervising these agents as opposed to writing the code yourself. In AI, 2027, one of the big accounts of how explosive AI growth might happen. That's one step, but then you need to take that engineering and use it to actually automate the AI research. You need to go from being able to implement the ideas more quickly to using that to fuel faster and faster growth in the breakthrough ideas themselves before you can turn that around and say, now make the AI better and better, at least in a really concerning way. You certainly go faster with just engineering. OpenAI talked about that with some of their launches from this past week, how the model played a role in this. Um, but it's not a full runaway train. There are also questions about what happens from there. Are there enough GPUs to go around? What bottlenecks might we encounter?

AI assessment note: “I think Matt's essay is directionally correct, but a bit early”

Q teams. The mission alignment team was created in twenty-twenty-four to promote the company's stated mission to ensure that artificial general intelligence benefits all of humanity. So of course, yeah, of course it makes sense to, you know, disband that one. Uh, they had also had, like, super alignment, which was also disbanded. Steven, you were close to this, uh, To this stuff. Uh, what is the, the implications on that?

A Seems pretty bad. Wish I were more surprised. Like, um, you know, at the end of 20, 24, which was when this team existed, open AI had announced plans to convert from a nonprofit to a for-profit in what seemed to me to be like pretty egregiously in violation of their commitments to the public. Um, and you know, they ended up Having to do a softer version of that because the attorneys general of California and Delaware got involved. So they didn't ultimately do something quite, quite so bad, but it's like, there's huge pressure on them. You know, they're planning to go public. Um, Josh who leads the team is a longtime friend of mine or who led the, the team, the mission alignment team is a longtime friend. I think really highly of him. I think that he sees the issues with AI very clearly. Um, You know, it does not surprise me that this is not quite so welcome at open AI any longer.

AI assessment note: “Seems pretty bad. Wish I were more surprised.”

Q do this. Okay, I'm actually going to go do this. And then it just shipped the code without me saying, go ahead and do this. Um, I, I do agree that we're getting to a point where the technology is getting much more powerful. Um, and you know, as for like this, you know, autonomous knowledge work, I'm not a hundred percent sure. Stephen, what do you think about that?

A Yeah, I think there's clearly been a change. I saw Kevin Roos joke on Twitter that his big AI policy idea is just get every Senator in a room and let them build their own website in, you know, 30 minutes with cloud code, something that they, they never could have done before. Um, the direction of travel seems very clear to me on this, like something has changed. More people are feeling the AGI in some sense, and I wouldn't want to mistake the The very excited tone of some of Matt's piece with meaning that the central claim is wrong. I think the central claim is right. It's just like a question of how soon we are going to get this form of displacement. And an unfortunate thing I think is people who are paying more to access the technology have this experience first. They kind of see what's coming and it's very, very easy to write that thing off as, oh, people are talking their own book. They're boosting their own companies. They want you to spend more money on AI and it's, it's just unfortunate, right? The AI you pay for. Is better, and it does help you feel this.

AI assessment note: “I think the central claim is right. It's just like a question of how soon”