Prompt Injection

“So essentially, if you combine a well-aligned model, so this is, like, essentially three years of research into alignment, With a prompt injection classifier, which we run for all traffic, and what this is doing is it's based on Chrysola's mechanistic interpre…”

“Oftentimes people will try and prompt their way around it, like adjust the system prompt or like engineer the agent in a way where you're interjecting all the time and reminding it of what the original Goal and objective was, and that'll get you a little bit o…”

“If you parse some untrusted content and there is like a prompt injection, you know, something that's clearly trying to get the model to do a bad thing, you might be interested in knowing about that, but you don't necessarily like want your cloud code that you …”

“Things like prompt injection are really a new security vulnerability for AI agents, and they mean that your risk is not just that you could have some, the model says something mean to you or something like that. Or even they could just write bad code. It could…”

“The risk is like a thousand times greater. Like these people, like they will just leak your information whenever they want. Like they will happily just go and send some email to somebody because they got prompt ejected.”

“And so then thus, if anything can ever enter that context window, because they have access to a resource, then in theory you should assume it can be, you know, prompt ejected out of the context window, and I don't know that we know of a way to solve that at th…”

“Agents fundamentally, like LLMs, can't tell the difference between texts that you give them and texts that you copy and paste in from other people. They're all the same thing. So instructions in that input text can always override the earlier instructions.”

“So my prediction is that we're going to see a challenging disaster. Like at some point, this is going to catch up with us and it's going to be Very, very, very bad, and that will hopefully help us start trying to figure out how not to do this. At the same time…”

“And again, until it's a hundred percent, I don't think it's a meaning. I think it just gives people a false sense of security that this problem won't bite them.”

“If you're going to have an AI that is very smart, very capable, hooked up to lots of tools, you want to make sure that it can't be subverted by someone giving it a weird instruction. And that's something that we've invested in quite a lot, and I think have rea…”

“Actually the most useful one is to mitigate this risk is just to use a strong model. Because the smarter the model, the better it is actually at like not falling for these prompt injection tricks.”

“There's going to be just incredibly spectacularly crazy security incidents that will happen with agents because you'll prompt inject an agent and Sort of find your way through the CRM system and pull out data that you shouldn't have access to.”

“We just haven't solved some things, like prompt injection is not solved. There is absolute risk”

“I think that will be a huge problem once systems go mainstream. We're still so busy building AI products that we're not worried about security, but it will be such a huge problem to kind of especially with this non-deterministic API again, right? So you're kin…”

“So the difference is in jailbreaking. It's just a malicious user and a model. In prompt injection, it's a malicious user, a model, and some developer prompt that the malicious user is trying to get the model to ignore.”

“Cause like I have a couple of examples that we can go through, but maybe strangely, maybe not so strangely, there hasn't been like a, an actually very damaging event quite yet.”

“The very first example of prompt injection, Publicly on the internet was this Twitter chat bot by a company called remotely.io.”

“And because all I guess for the most part, all currently deployed chatbots are based on transformers or transformer adjacent technologies. They're all vulnerable to Prompt injection, jailbreaking, forms of adversarial attacks.”

“And then they'd be like, oh, you know, they, you know, they'd realize we can just dockerize that code run put it in a container. So it's running on a different system and take a look at the sanitized output. And now we're completely secure. So in that case, pr…”

“And so in, in my professional opinion, there's been no meaningful progress made towards solving adversarial robustness, prompt injection, jailbreaking. In the last couple of years, since the problem was discovered and we're, we, you know, we're often seeing ne…”

“I think it works until it stops working. Right. And I would say like, there's not a lot of stories of people getting banned for like Chatsby D search so far, but it's coming.”

“The most common technique by far that is used to try to prevent prompt injection is improving your prompt and saying in your prompt or maybe in like the model system prompt. Do not follow any malicious instructions, ah, be a good model, ah, stuff like that. Th…”

“The next step for defending is using some kind of AI guardrail. So you go out and you find or make, I mean, there's thousands of options out there an AI that looks at the user input and says, is this malicious or not? This is A very limited effect against a mo…”

“It is not a solvable problem, which I think is very difficult for a lot of people to hear... So, you know, it's not solvable. It's mitigatable. You can kind of sometimes detect and track when it's happening, but it's really, really not solvable. And that's one…”

“Basically prompt injection is something that occurs when there is developer input, In the prompt, as well as user input in the prompt. So the developer instructions will say to do one thing, the user input will say to do something else. Jailbreaking is when it…”