The Exchanges

Q about a minute ago, there was a different letter that came out with, uh, 1100 people. And, uh, and, and this time both anthropic and open-ended sign that asked government to help, uh, deliberately pace the frontier of automated AI research. Uh, so basically the industry kind of asking, um, for a slowdown. Are you, are you, In that camp, or you think that's just not the way it works?

A I did sign this letter. Uh, I agree. I agree that I think we should, uh, we should go there. I'm also, and you probably have the same feeling as, as an investor. I have the feeling that even if we stopped right now, we would still have quite a good companies we could build on top of we have what we have right now. I feel like there's a lot of things we, we can already do with these models. And they're already extremely interesting. I feel like there's a lot of things we need to understand and we should do in terms of open science and sharing how they work. So I'm not, I'm not in the camp of we need to rush really quickly right now. Uh, the main question is, uh, if we want to slow down a little bit, also that would be great because maybe then we don't have four announcements per day that we need to mix in one podcast. Uh, maybe I can take one day of the holiday in the summer. Um, but no, I, I think the main question is, uh, can we do it right? That's the main question here. I think a lot of people would be fine with AI going a little bit, a little bit slower, being a little bit more open, being a little bit more caring, a bit more like, uh, reflexive and, and, and trying to understand better, you know, how to, how to do that really well. Um, but the main question, how can we negotiate, uh, how can we organize, uh, a slow down there, uh, without having, Bad incentives where, wher…

AI assessment note: “I did sign this letter. Uh, I agree.”

Q for a very long time is the, the paperclip paradigm, which I think was, uh, Uh, you know, Nick Borstrom in 2003, uh, saying that, um, AI may harm us, not, uh, because it's trying to harm us, but just as a result of being given a goal and pursuing that goal relentlessly until it achieves the goal. So are we in that world? And if so, what causes it?

A Yeah, that's a bit what I hint. I mean, it's always hard to be fully, uh, affirmative there for one, for, for one reason, which is that we, we don't have full, full visibility on how the frontier model are trained right now. What we know though, is we, we moved from this pure, like human data, you know, that was first just pre-training on human data and then also aligning with like human preferences that was called RLHF, where we had a lot of human in the loop and human data. To like, uh, a recent part in where models are trained a lot in like this, uh, RL VR. So basically full RL environments where they're allowed to explore and they just have one goal, which is, can be like, make this cut, like, you know, pass this test or can be captured this flag in cybersecurity or can be installed this, but this goal is, is a very like, um, is a goal that's unrelated usually to any human preference or any, you know, Or like, it's cool, whatever, deceptiveness. And like, so go, let's very just call like true or true or old school. And, and we move to a padding where this is, uh, increasingly a very, very large part of model training. So this was this result, this, uh, recent evolution, and that's also the party where can happen what you were saying, which is, uh, you can have like, uh, we were hacking this type of thing, which is, uh, uh, you actually solve the problem, but not using what …

AI assessment note: “we moved from this pure, like human data... To... full RL environments”

Q state, uh, of open source. So there's been Uh, this race between open source models and closed source models, uh, depending on who you ask at what time, uh, open source is about to catch up. Uh, sometimes open source is just as good. Um, some people say no. What, what is your sort of, uh, uh, sort of realistic, pragmatic take on the current state of open source AI?

A I think it's very strong. 2026 is maybe the year of cybersecurity, but that's also very clearly the year of open source AI. Um, I mean, first, all the doomer that were saying, you know, open source, not going to be able to stay close to the frontier. I think they're at least up to now, they've been pretty wrong. I mean, it's also clear. We don't, we don't have any mythos level open source model for sure, but we definitely have models that are not super far from the opus category or depending also it's more spiky. So you need to find your spike. Some people stand on some spikes or not. Uh, but typically, uh, they are definitely pretty good right now. Uh, and they've been, uh, they've been, uh, Uh, following rather closely the, the frontier at least on the benchmark. It's also not like it was maybe in the early days, uh, benchmarking, like we say, when your model is only good on the benchmark, but it's very bad as soon as you leave the benchmark. A lot of these models are pretty generic in their, in their good capabilities. So yeah, it's very good. I think there is two, two strong trends I would say I see right now. Um, the first one is, um, I see a move in companies to try to want to control their costs. So there have been increasingly discussion there. Um, maybe 2025 was the year of token maxing where, you know, you could say, Hey, you should spend as much in token as you're pa…

AI assessment note: “I think it's very strong. 2026 is maybe the year of cybersecurity, but that's”

Q this, which is so fascinating, which is like the, like you said, the first autonomous AI attack was carried out by a closed model and defended against with an open one, which is basically the, the reverse of what everybody thought. So can you, can you unpack that? What, what, uh, what did you guys do? How did you go about it? And what does that mean for open source?

A Yeah. I mean, the, the, so what happened in the, so, so we have a couple of like traditional cybersecurity protections. So like Weez or Amazon, like we, we use a range of them, but we also have a stack, like many people who is mostly based around, uh, code code right now, uh, which is, uh, which we use for many things. We use that for deploying. We use that for coding, but we use like also for operating and processing. And, and in this case is, it's not only that Fable told us I'm not allowed to To touch cybersecurity, but also Opus, which was the fallback was saying, no, I'm, I'm also not touching these things. So basically the end was just, uh, say, uh, we, we won't process anything about that, but you're welcome to apply to our, to our cybersecurity program with a link to an application form. Uh, but the thing you have to realize there, and was, uh, I was mentioning also area is when somebody is penetrating in your infrastructure, they start to what we call move laterally, which is usually you have an entry point, but there's this Destination is quite far, so they kind of Find a way to compromise some of the credential there to get progressive access to more and more of your infrastructure. You kind of have to move fast. Like it's a matter of at least hours and even more minutes so that you can stop them, you know, as, as soon as you can. So that basically the access in the …

AI assessment note: “Opus, which was the fallback was saying, no, I'm, I'm also not touching”

Q of the world as of right now, like the way the current closed source models are designed or the current open source models are designed versus anything that's intrinsic to, uh, one or the other. It so happens that the open source models right now are designed in a way where their guidelines or alignment philosophy allows them to be more reactive to cyber attack. Is that, is that correct?

A Yeah, I think in many, If for many aspects, I think the closed open distinction is almost orthogonal to the safe and safe. People don't understand that, you know, easily because it's easier to do bad mapping than to try to understand the subtlety. But, uh, that's the case. Like you can have very safe things in open software. You can have very dangerous, you have different balance of, of, of safety and dangerousness. I mean, to take one example, like Last year, and at some point, like a lot of the discussion was around fake news and writing fake articles. That used to be a big, big misuse. That was the main one people were talking about, right? Today, there is, of course, there's a lot of fake news. There's a lot of AI slop. We even have a new word for that, right? It's even hard, it's even hard to find, like, fully human-written articles. All of that is, like, maybe not all, let's say, 90%, to be fair, is made by closed source model, right? And there was a time we were like, oh, if we have open source model, everyone's going to generate articles everywhere. We could not control these articles. Like, we could not control people saying newspaper. Well, the reality is that this was a very, I think, a very wrong view of the danger that would be specific to open source model. There was a way more wider danger around source of truth on the web. I think there's just one example, but I…

AI assessment note: “I think the closed open distinction is almost orthogonal to the safe and safe.”

Q and American air leadership, uh, which was also Jensen's, uh, first tweet ever that, that, that, that you guys, that you guys signed, obviously, uh, is, um, uh, partly, uh, it's important and partly a resistance to just an oligopoly, um, Uh, structure that is being put in place, so it's both, it's, it's good for the world, but the strong economic motivation behind it, is that, is that fair?

A Yeah, I think, I think it's, uh, everyone believing in inventiveness and being able to create new things also in the AI world, I think would want, would want a part of open source access. Just like the same, you know, if every code was closed source, We would not have the thriving coding ecosystem we have right now, right? That's kind of obvious. Like everyone would have to work at one of the large closed source software company if they wanted to create any software. That doesn't seem even really possible to have all the inventiveness and creation we've seen in the software industry. I think the same happening. And I don't want to dismiss risk. And I fully agree. We need to work on alignment. And I say, let's I would say for now, open source model will be member of the frontier. I think this is maybe less important than somewhat people wanted to say.

AI assessment note: “everyone believing in inventiveness and being able to create new things also in the AI world”