Emmons: Prompt guardrails fail to reliably constrain AI agents
Shane Emmons · Building Trustworthy AI: Navigating the Challenges and Future of Agentic Software with Shane Emmons · Nov 4, 2025 · at 9:37
Shane Emmons, CEO of Swept, explains to Christian Marquez why prompt-based AI guardrails fail to prevent catastrophic failures without external hardline enforcement.
0:00 / 0:37exact quote · 37.2s
720p mp4 · rendered on demand · StarZero watermark
“So stepping in there with policy that says do not go on this, that's a big deal for things in the medical setting where you may have, you know, people use the word guardrails to talk about AI where they're like, we're going to put guardrails so it doesn't do these things. Guardrails can still be smashed through by semis and, you know, jumped by people. And so that's where we come in and actually put hardline policy that just because you told the agent you can't prescribe more than 100 milligrams of this drug doesn't mean it won't occasionally choose that for whatever reason. And so you want to have hardline policy in place that stops it.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Shane Emmons
Emmons: Secure underlying APIs instead of trusting AI tool calls
“If somebody can ask the agent to make arbitrary tool calls, you really just need to make sure that those APIs and systems it's calling are properly secured versus trusting the AI is only going to make the right calls there.”
Emmons: AI ROI comes from agents doing work, not handbook copilots
“And that's where they're bringing ROI. It's not in co-pilots to ask information about your handbook. It's about agents actually doing work for your team.”
Emmons: Requiring humans to review every AI output eliminates ROI
“If a whole, if a human's now tasked with reviewing everything, we find a lot of times say you're doing a claims adjustment or underwriting an insurance place capacity, you would have where you're not making the value back because the person that's checking it …”
Emmons: AI's Probabilistic Imperfection Is a Benefit, Not Just a Limitation
“AI is meant for predicting things where we don't have perfect knowledge, and so that's just a fundamental, you can see it as a limitation, but you also can see it as really, I think of it as a benefit of AI, because there are all kinds of algorithms that We co…”
Emmons: Test AI at the scoped agent level, not foundation models
“When we're testing them and trying to figure out how they function, one of the first things we do is we don't go down to the model level, the chat GPDs, the clods, the Geminis of the world, because they have You can think of it as, like, infinite possibility. …”
Emmons: Hackers Will Continually Bypass AI Guardrails Despite Future Breakthroughs
“We may have, and I would think we'll have big breakthroughs over the coming years that really help curb some of this, but there's still going to be the black and white hat hackers who are going to try to prove that we can get around it just like we have today,…”