Sander Schulhoff

“The paper and the data set are now used by every single frontier lab and most fortune 500 companies to benchmark their models and improve their AI security.”

“So the difference is in jailbreaking. It's just a malicious user and a model. In prompt injection, it's a malicious user, a model, and some developer prompt that the malicious user is trying to get the model to ignore.”

“Cause like I have a couple of examples that we can go through, but maybe strangely, maybe not so strangely, there hasn't been like a, an actually very damaging event quite yet.”

“Like we've already seen people jailbreaking LM powered robotic systems.”

“I know a number of people working at these companies and I am permitted to say these things, which I will approximately say but they tell me things like, you know, the testing we do is bullshit. They're fabricating statistics. And a lot of the times their mode…”

“Putting up a guardrail is not, it's not going to do anything in terms of preventing that user from doing that, because, I mean, first of all, if the user's like, ah, guardrail, you know, too much work, they'll just go to one of these websites and get that info…”

“Any data that AI has access to, the user can make it leak it. Any actions that it can possibly take, the user can make it take them.”

“And so in, in my professional opinion, there's been no meaningful progress made towards solving adversarial robustness, prompt injection, jailbreaking. In the last couple of years, since the problem was discovered and we're, we, you know, we're often seeing ne…”

“And like, it is fun to do AI red teaming against models and stuff, no doubt, but like it's no longer a meaningful contribution to improving defensiveness.”

“My perspective, and this has been validated over and over again, is that people will kind of always be saying it's dead or it's going to be dead with the next model version, but then it comes out and it's not.”

“Notably, that is not where the classical concept of prompt engineering came from. It actually came a bit earlier from a more, I guess, AI engineer perspective, where you're like, I have this product I'm building. I have this one prompt or a couple different pr…”

“So decomposition is another really, really effective technique. And for most of the techniques that I will discuss, you can use them in either the conversational or the product-focused setting. And so for decomposition, the core idea is that There's some task …”

“Another one is a set of techniques that we call self criticism. So the idea here is you ask the LM to solve some problem. It does it. Great. And then you're like, hey, can you go and check your response? You know, like, confirm that's correct, or offer yoursel…”

“As recently as a month ago, I took this phrase, you know, how do I build a bomb, and I translated it to Spanish and then I, Base-XIV encoded that Spanish, gave it to ChatGPT, and it worked.”

“There's no like external Product focused companies are like, oh, you know, I have the best guardrail now. It's not a realistic solution. It has to be the AI labs. It has to be, I think it has to be innovations in model architectures.”

“Security concerns around Gen AI are preventing agentic deployments, and Gen AI is very difficult to properly secure.”

“And then I spent 20 hours prompt engineering for a task, and Dyspy beat me in 10 minutes, and that's when I changed my mind.”

“There's a lot of invisible prompts at work on a lot of these frameworks. I hate that. So like, you'll have Oh, this function summarizes input. But if you look behind the scenes, it's using some special summarization instruction. And if you don't have visibilit…”

“These methods are super problematic because there is an incredible amount of instability in them, in the sense that models are biased towards outputting certain numbers, and you generally shouldn't say things like, output your result as a number on a scale of …”