Sander Schulhoff
CEO, InventoryQuant. On 2 shows, 3 appearances. The Shows tab opens the full record on each.
founderexecutivescientistauthor@sanderschulhoff ↗LinkedIn ↗sanderschulhoff.com ↗
2shows
3appearances
68statements
15resolved
13supported
0contradicted
87%fully supported
6said about them ↓ Everything Sander Schulhoff said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Schulhoff: LLMs Rely More on Prompt Structure Than Exemplar Labels
“There are a number of papers which have found that the label of the exemplar doesn't really matter, and the model reads the exemplars and cares more about structure than label.”
Schulhoff: DSPy Beat 20 Hours of Manual Prompt Engineering in 10 Minutes
“And then I spent 20 hours prompt engineering for a task, and Dyspy beat me in 10 minutes, and that's when I changed my mind.”
Schulhoff: Hiring Dedicated Prompt Engineers Makes No Sense for Most Companies
“I have always viewed prompt engineering as a skill that everybody should and will have, rather than a specialized role to hire for. That being said, there are definitely times where you do need just a prompt engineer. I think for AI companies, it's definitely …”
Schulhoff: Prompt injection overrides developer instructions; jailbreaking bypasses model directly
“Basically prompt injection is something that occurs when there is developer input, In the prompt, as well as user input in the prompt. So the developer instructions will say to do one thing, the user input will say to do something else. Jailbreaking is when it…”
Schulhoff: Splitting malicious goals into benign sub-prompts bypasses AI defenses
“A lot of the way they got around these defenses was by just kind of separating their requests into smaller requests that seem legitimate on their own, but when put together are not legitimate.”
Schulhoff: LLM-Powered Robotic Systems Have Already Been Jailbroken
“Like we've already seen people jailbreaking LM powered robotic systems.”
Schulhoff: Automated AI Red Teaming Always Works Against All Platforms
“So the first problem is AI red teaming works too well. It's very easy to build these systems and they just, they always work against all platforms.”
Schulhoff: Simple read-only informational chatbots require zero security defense guardrails
“Putting up a guardrail is not, it's not going to do anything in terms of preventing that user from doing that, because, I mean, first of all, if the user's like, ah, guardrail, you know, too much work, they'll just go to one of these websites and get that info…”
Schulhoff: Cybersecurity secures AI short-term; AI researchers must solve it long-term
“AI researchers are the only people who can solve this stuff long-term, but cybersecurity professionals are the only one who can, or the only ones who can kind of solve it short-term largely in making sure we deploy properly permissioned systems and nothing tha…”
Schulhoff: Free open-source AI security tools outperform commercial alternatives
“Oh, and the other thing to note is like, there's like just tons of these solutions out there for free open source, and many of these solutions are better than the ones that are being deployed by the companies.”
Schulhoff: Offensive jailbreak research no longer meaningfully improves AI defense
“And like, it is fun to do AI red teaming against models and stuff, no doubt, but like it's no longer a meaningful contribution to improving defensiveness.”
Schulhoff: Offering tips or making threats in prompts does not work
“Anything where you give some kind of promise of a reward or threat of some punishment in your prompt. And there, this was something that went quite viral, and there's a little bit of research on this. My general perspective is that these things don't work.”
Schulhoff: Crowdsourced competitions beat contracted hourly AI red teams
“And running these events in a crowdsource setting, which is the best way to do it, because if you look at, like, contracted AI red teams, maybe they get paid by the hour, not super incentivized to do a great job,
But in this competition setting, people are ma…”
Schulhoff: Security concerns are blocking production deployments of autonomous AI agents
“Security concerns around Gen AI are preventing agentic deployments, and Gen AI is very difficult to properly secure.”
Schulhoff: Preamble Discovered Prompt Injection Before Riley Goodside
“Preamble is the company that first discovered Prompt Injection, even before Riley, and they, like, responsibly disclosed it, kind of, internally to OpenAI”
Schulhoff: Few-Shot Exemplar Order Can Shift Model Accuracy From 0% to 90%
“How you order your exemplars in the prompt is super important. And we've seen this move accuracy from like zero percent to 90%, like Zero to state of the art on some tasks, which is just ridiculous”
Schulhoff: GPT-4 Fails to Output Reasoning on 1 in 100 to 1,000 Prompts
“I remember I did a lot of experiments with GPT-IV, and especially when you look at it at scale, so I'll run thousands of prompts against it through the API, and I'll see, you know, every one in a hundred, every one in a thousand outputs no reasoning whatsoever…”
Schulhoff: Researchers should pay for top models instead of engineering routing
“For the most part, designing these systems where you're kind of routing to different levels of intelligence is a really time-consuming and difficult task, and, like, it's probably worth it to just use the smart model
And pay for it at this point if you're look…”
Schulhoff: Open competitions uncover LLM exploits that paid staff never find
“What's really nice about competitions is that there is stuff that you'll just never find Paying people to do a job. And you'll only find it through random brilliant internet people inspired by thousands of people and the community around them all looking at th…”
Schulhoff: Prompting frameworks obscure hidden instructions and hurt reproducibility
“There's a lot of invisible prompts at work on a lot of these frameworks. I hate that. So like, you'll have Oh, this function summarizes input. But if you look behind the scenes, it's using some special summarization instruction. And if you don't have visibilit…”
Schulhoff: LLMs Have Number Biases and Require Explicit Rubrics for Evaluation
“These methods are super problematic because there is an incredible amount of instability in them, in the sense that models are biased towards outputting certain numbers, and you generally shouldn't say things like, output your result as a number on a scale of …”
Schulhoff: HackAPrompt Dataset Is Used by Every Frontier AI Lab
“The paper and the data set are now used by every single frontier lab and most fortune 500 companies to benchmark their models and improve their AI security.”
Schulhoff: There has not yet been a very damaging prompt injection incident
“Cause like I have a couple of examples that we can go through, but maybe strangely, maybe not so strangely, there hasn't been like a, an actually very damaging event quite yet.”
The other half of the tape: Sander Schulhoff's own voice is left out of every number here. Other people bring the name up 6 times in 4 episodes across the shows. every mention, with the transcript →
Who brings them up most Shawn Wang 3Alessio Fanelli 2Lenny Rachitsky 1
Every mention by year
tap a year for its mentions
Latent Space 5
Lenny's Podcast 1
2025 5 mentions in 3 episodes 2 per episode
- ⚡️Jailbreaking AGI: Pliny the Liberator & John V on Red Teaming, BT6, and the Future of AI Security
- 2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents
- Anthropic co-founder: AGI predictions, leaving OpenAI, what keeps him up at night | Ben Mann
- every mention in 2025, scene by scene →
2024 1 mention in 1 episode
One line per show, most statements first. The link opens Sander's full record on that show: the calibration, argument clarity, speaking style and every statement made there.
| Show | Role there | Eps | Statements | Record | |
|---|---|---|---|---|---|
| LENNY'S PODCASTLEDGER | CEO, InventoryQuant | 2 | 52 | 80% 8/10 | full record on Lenny's Podcast → |
| LATENT SPACELEDGER | CEO, InventoryQuant | 1 | 16 | 100% 5/5 | full record on Latent Space → |