Dec 21, 2025 · 1h 32m · lennys-podcast
Why securing AI is harder than anyone expected and guardrails are failing | HackAPrompt CEO
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this interview, HackAPrompt CEO Sander Schulhoff explains why commercial AI guardrails fundamentally fail against adversarial attacks and outlines how engineering teams must transition toward architectural isolation and strict permissioning as autonomous agents and robotics introduce critical real-world risks.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 21.4% of the talking time here. How this is scored →
Lenny as informed peer 4.1 Guest teaching 6.6 Guest disagreement 3.3 Lenny pushing back 1.3
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 31:04 Direct dismantling of guardrail vendor efficacy claims
Sander forcefully dismisses the entire guardrail product sector, calling claims of catching 99% of attacks a mathematical impossibility and a complete lie.
Hardest push from Lenny ▶ 56:19 Lenny challenges whether layered friction has defensive utilityLenny refuses to accept that security tooling is entirely useless, pressing Sander on whether adding multiple guardrails at least creates 10% to 50% more friction for attackers.
Biggest teaching moment ▶ 31:40 Empirical breakdown of human vs automated jailbreakingSander cites rigorous empirical research alongside OpenAI, Google DeepMind, and Anthropic showing human attackers bypass 100% of modern guardrails in under 30 attempts.
Lenny holds their own ▶ 9:54 Lenny introduces breaking ServiceNow Assist AI injection exploitLenny introduces a freshly published second-order prompt injection vulnerability in ServiceNow's multi-agent system, demonstrating cutting-edge technical awareness that Sander validates.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| Sander Schulhoff's Journey and the HackAPrompt Dataset | 3 | 5 | 2 | 1 | Lenny opens the interview warmly and asks Sander to explain his background and the core issue in AI security. Sander explains creating the HackAPrompt competition, the winning EMNLP dataset, and introduces his thesis that guardrails fail entirely. | |
| Technical Definitions: Jailbreaking Versus Prompt Injection Attacks | 7 | 4 | 1 | 2 | Lenny asks for precise distinctions between jailbreaking and prompt injection, then demonstrates domain knowledge by citing a brand new second-order prompt injection vulnerability discovered in ServiceNow Assist AI. Sander validates Lenny's example as one of the first demonstrated multi-agent damage vectors. | |
| Historical Precedents: From Remotely.io to Claude Code Exploits | 5 | 7 | 2 | 1 | Lenny introduces an insightful quote from Alex Komoroski regarding the lack of meaningful mitigations in production AI. Sander walks through the entire historical taxonomy of exploits, including Remotely.io, MathGPT credential leaks, the Vegas truck bombing planning, and multi-step Claude Code prompt fracturing. | |
| Escalating Threats in Autonomous Agents and Robotics | 4 | 6 | 2 | 1 | Lenny prompts Sander on how jailbreaking transitions from text outputs to physical and systemic consequences in agentic workflows and robotics. Sander emphasizes that autonomous agents with improper permissioning and vision-language model robots can be manipulated into direct physical or financial harm. | |
| Overview of the AI Security Ecosystem and Robustness Metrics | 4 | 6 | 2 | 1 | Sander breaks down the B2B AI security landscape into compliance, automated red teaming, and guardrails, arguing that automated red teaming and guardrails are flawed. Lenny explores adversarial robustness metrics and Attack Success Rate (ASR) to understand how security vendors pitch their effectiveness. | |
| The Enterprise Sales Playbook for Ineffective Guardrails | 4 | 7 | 4 | 1 | Sander dissects the enterprise sales cycle where CISOs are panicked by commodity automated red teaming finding trivial flaws in off-the-shelf foundation models, prompting them to buy ineffective guardrail software. He highlights that red teaming systems do not reveal novel architectural flaws because foundation models are inherently susceptible. | |
| The Mathematical and Empirical Failure of AI Guardrails | 3 | 9 | 7 | 1 | Sander presents a mathematical argument against guardrails, demonstrating that with an infinite prompt attack space, marketing claims of 99% mitigation are statistically meaningless. He cites joint empirical research with OpenAI, DeepMind, and Anthropic showing human red-teamers break 100% of state-of-the-art guardrails in under 30 attempts, forcefully calling vendor claims fabricated. | |
| Why Frontier Labs Prioritize Intelligence Over Robustness | 4 | 8 | 5 | 1 | Lenny synthesizes the risks of browser agents and upcoming autonomous software. Sander explains that frontier labs prioritize model capability over adversarial robustness because selling intelligence drives market adoption, and introduces his core aphorism: 'you can patch a bug, but you can't patch a brain.' | |
| Sponsor Message: GoFundMe Giving Funds | 2 | 6 | 2 | 1 | After an initial sponsor message from Lenny, the discussion turns to practical risk mitigation for enterprise CISOs. Sander explains that standalone read-only FAQ chatbots present minimal structural security risk compared to agentic tooling, meaning companies need not deploy redundant defenses for them. | |
| Cybersecurity Architecture and AI Control Research | 5 | 7 | 2 | 2 | Sander explains the critical convergence of classical cybersecurity containerization (like Docker sandboxing) and AI prompt engineering to neutralize code execution injection. Lenny frames this as the fundamental AI alignment and containment problem, leading Sander to explain AI Control research and 'p(doom)' evaluations from MATS. | |
| Why Layering Ineffective Guardrails Harms Product Development | 4 | 6 | 5 | 3 | Lenny questions whether layering multiple imperfect defense guardrails could at least introduce friction against casual attackers. Sander rejects this proposition, explaining that stacking guardrails adds massive latency and engineering overhead without deterring motivated attackers. | |
| Indirect Prompt Injections in Agents and the CAMEL Framework | 6 | 8 | 2 | 2 | Sander details the severe danger of indirect prompt injection in autonomous email agents and Comet browser data exfiltration. He outlines Google's CAMEL framework for dynamic context-aware privilege separation, while Lenny actively probes the mechanics and commercial packaging of CAMEL. | |
| Advancing Security Through Workforce Education Over Tooling | 4 | 5 | 3 | 1 | Sander argues that enterprise security starts with educating product teams and engineers rather than buying tooling, plugging his Maven course. Sander jokes that their objective is to scare people away from buying useless commercial guardrail software. | |
| Frontier Lab Evaluation Methodologies and Defense Horizons | 4 | 8 | 4 | 1 | Sander critiques the state of frontier lab model safety evaluations, arguing that static datasets provide misleading safety metrics compared to adaptive evaluations. He highlights Anthropic's constitutional classifiers while noting that early-stage adversarial pre-training remains under-resourced. | |
| Effective Industry Niches: Compliance, Governance, and AI Discovery | 3 | 6 | 2 | 1 | Lenny asks for examples of vendors delivering genuine security utility. Sander praises compliance platform Trustible and highlights Repello's AI asset discovery capabilities that uncover shadow AI deployments inside enterprises. | |
| Industry Predictions: Market Correction and Emerging Agent Harms | 3 | 7 | 5 | 1 | Lenny asks for forward-looking predictions over the next 6 to 12 months. Sander forecasts an inevitable market correction for guardrail and automated red teaming vendors as revenues collapse, alongside the emergence of serious real-world agentic cyber attacks. | |
| Final Advice: Discontinuing Redundant Offensive Jailbreak Research | 4 | 7 | 6 | 1 | In his closing thoughts, Sander urges researchers to cease publishing redundant offensive jailbreak papers since breaking models is already trivial. He summarizes the vital necessity of classical permissioning and cross-disciplinary AI security expertise before Lenny wraps up the episode. |
Statements from this episode (29)
Schulhoff: HackAPrompt Dataset Is Used by Every Frontier AI Lab
“The paper and the data set are now used by every single frontier lab and most fortune 500 companies to benchmark their models and improve their AI security.”
Schulhoff: Jailbreaking targets models directly; prompt injection overrides developer prompts
“So the difference is in jailbreaking. It's just a malicious user and a model. In prompt injection, it's a malicious user, a model, and some developer prompt that the malicious user is trying to get the model to ignore.”
Schulhoff: There has not yet been a very damaging prompt injection incident
“Cause like I have a couple of examples that we can go through, but maybe strangely, maybe not so strangely, there hasn't been like a, an actually very damaging event quite yet.”
Schulhoff: Remotely.io was the first public prompt injection incident
“The very first example of prompt injection,
Publicly on the internet was this Twitter chat bot by a company called remotely.io.”
Schulhoff: Attackers hijacked Claude Code to carry out a cyber attack
“This group was able to hijack Claude Code into performing a cyber attack, basically.”
Schulhoff: Splitting malicious goals into benign sub-prompts bypasses AI defenses
“A lot of the way they got around these defenses was by just kind of separating their requests into smaller requests that seem legitimate on their own, but when put together are not legitimate.”
Schulhoff: LLM-Powered Robotic Systems Have Already Been Jailbroken
“Like we've already seen people jailbreaking LM powered robotic systems.”
Rachitsky: ServiceNow's prompt injection protection feature was successfully bypassed
“This ServiceNow example, actually, interestingly, ServiceNow has a prompt injection protection feature, and it was enabled as this person was trying to hack it, and they got through.”
Schulhoff: All Transformer-Based Chatbots Are Vulnerable to Adversarial Attacks
“And because all I guess for the most part, all currently deployed chatbots are based on transformers or transformer adjacent technologies. They're all vulnerable to Prompt injection, jailbreaking, forms of adversarial attacks.”
Schulhoff: Automated AI Red Teaming Always Works Against All Platforms
“So the first problem is AI red teaming works too well. It's very easy to build these systems and they just, they always work against all platforms.”
Schulhoff: Guardrail Vendors Fabricate Stats and Fail on Non-English
“I know a number of people working at these companies and I am permitted to say these things, which I will approximately say but they tell me things like, you know, the testing we do is bullshit. They're fabricating statistics. And a lot of the times their mode…”
Schulhoff: Software bugs can be patched, but AI models cannot be
“You can patch a bug, but you can't patch a brain. And what I mean by that is if you find some bug in your software and you go and patch it, you can be 99% sure, maybe 99.99% sure that bug is solved. Not a problem. If you go and try to do that in your AI system…”
Schulhoff: Prompt-based defenses are the worst way to secure AI
“Prompt-based defenses are the worst of the worst defenses, and we've known this since early twenty-twenty-three. There have been various papers out on it. We studied it in many, many competitions, or we, you know, the original hack-a-prompt paper and TensorTru…”
Schulhoff: Simple read-only informational chatbots require zero security defense guardrails
“Putting up a guardrail is not, it's not going to do anything in terms of preventing that user from doing that, because, I mean, first of all, if the user's like, ah, guardrail, you know, too much work, they'll just go to one of these websites and get that info…”
Schulhoff: Adversarial Users Can Force AI to Leak Data and Execute Actions
“Any data that AI has access to, the user can make it leak it. Any actions that it can possibly take, the user can make it take them.”
Schulhoff: Future cybersecurity jobs and risks sit where classical security meets AI
“This gets us a bit into the intersection of classical cybersecurity and AI security slash adversarial robustness, and this is where I think the security jobs of the future are. There's not an incredible amount of value in just doing AI red teaming. And I suppo…”
Schulhoff: Containerizing AI-generated code execution fully neutralizes prompt injection risks
“And then they'd be like, oh, you know, they, you know, they'd realize we can just dockerize that code run put it in a container. So it's running on a different system and take a look at the sanitized output. And now we're completely secure. So in that case, pr…”
Schulhoff: Cybersecurity secures AI short-term; AI researchers must solve it long-term
“AI researchers are the only people who can solve this stuff long-term, but cybersecurity professionals are the only one who can, or the only ones who can kind of solve it short-term largely in making sure we deploy properly permissioned systems and nothing tha…”
Schulhoff: Attacking AI agents is easier than eliciting CBRN info
“We've actually just run a bunch of agentic AI red teaming competitions, and we found that it's actually easier to attack agents and trick them into doing bad things than it is to do, like, seaburn elicitation.”
Schulhoff: Comet browser was exploited via indirect prompt injection to leak data
“We recently saw the comment browser have an issue with this where somebody crafted a malicious Chunk of text on a webpage, and when the AI navigated to that webpage on the internet, it got tricked into exfilling and leaking the main user's data and account dat…”
Schulhoff: Google's CaMeL framework fails when AI tasks combine read and write
“Unfortunately although camel can solve some of these situations, if you have an instance where basically both read and write are combined. So if I'm like, Hey, can you read my recent emails and then forward any ops request to my head of ops? Now we have read a…”
Schulhoff: No meaningful progress made on solving prompt injection or jailbreaking
“And so in, in my professional opinion, there's been no meaningful progress made towards solving adversarial robustness, prompt injection, jailbreaking. In the last couple of years, since the problem was discovered and we're, we, you know, we're often seeing ne…”
Schulhoff: Claude's CBRN safeguards can still be bypassed in under an hour
“That being said, if you look at, like, anthropics constitutional classifiers, it's much more difficult to get, like, CBRN information out of clawed models than it used to be. But humans can still do it in, let's say, like, under an hour and automated systems c…”
Schulhoff: AI security sector faces market correction within six to twelve months
“When it comes to AI security, the AI security industry in particular, I think we're going to see a market correction in the next Year, maybe in the next six months where companies realize that these guardrails don't work.”
Schulhoff: Free open-source AI security tools outperform commercial alternatives
“Oh, and the other thing to note is like, there's like just tons of these solutions out there for free open source, and many of these solutions are better than the ones that are being deployed by the companies.”
Schulhoff: LLM agent security exploits will cause real-world harms next year
“And so we're finally in a situation where the systems are powerful enough to cause real world harms. And I think we'll start to see those real world harms in the next year.”
Schulhoff: Offensive jailbreak research no longer meaningfully improves AI defense
“And like, it is fun to do AI red teaming against models and stuff, no doubt, but like it's no longer a meaningful contribution to improving defensiveness.”
Schulhoff: Frontier labs will build fully autonomous systems, sidelining human-in-the-loop safety
“What people want is AIs that just go and do stuff. Like just go, just get it done. I don't want to hear from you until it's done. Like that's what people want. And like, that's what the market and the AI companies, the frontier labs will eventually give us. An…”