Sep 20, 2024 · 1h 7m · latent-space
The Ultimate Guide to Prompting - with Sander Schulhoff from LearnPrompting.org
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Sander Schulhoff, creator of LearnPrompting.org and lead author of The Prompt Report, joins the Latent Space podcast to unpack the empirical science of prompt engineering, adversarial AI safety, and the programmatic evolution toward AI engineering.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 27.2% of the talking time here. How this is scored →
The hosts as informed peer 6.4 Guest teaching 4.0 Guest disagreement 1.7 The hosts pushing back 3.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 16:40 Dismissing role prompting efficacy for accuracy tasks
Sander rejects mainstream assumptions about role prompting, pointing out his empirical benchmark where prompting a model as an idiot outperformed prompting it as a Harvard math professor on MMLU.
Hardest push from the hosts ▶ 43:54 Swyx proves the necessity of cost-aware model cascadesSwyx directly counters Sander's dismissal of model routing by citing specific pricing figures ($5/M tokens vs $0.15/M tokens) to demonstrate why ensembling cheap models under a smart judge is essential for production.
Biggest teaching moment ▶ 50:53 The formal boundary between prompt injection and jailbreakingSander unpacks the widespread confusion in industry and literature between prompt injection (developer instructions overridden by user data) and jailbreaking (user directly bypassing safety guardrails).
The host holds their own ▶ 19:31 Swyx defending multi-persona reasoning and synthetic generationSwyx counters Sander's critique of role prompting by citing concrete research examples including Salesforce's DEI benchmark and Tencent's billion-persona synthetic data paper.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Sander Schulhoff's Background and Journey to The Prompt Report | 3 | 1 | 0 | 0 | The episode opens with standard biographical interviewing where Swyx and Alessio prompt Sander to narrate his progression from Diplomacy and MineRL reinforcement learning to founding LearnPrompting and leading the Hack-A-Prompt and Prompt Report research initiatives. Sander delivers an extensive monologue detailing his academic path and research publications with minimal interruption. | |
| Systematically Reviewing Academic Literature via the PRISMA Process | 6 | 4 | 1 | 1 | Swyx demonstrates strong literacy regarding current AI literature generators like Sakana AI and Omniscience, prompting a discussion on systematic literature reviews. Sander explains the rigorous PRISMA methodology and how his team benchmarked LLMs against human evaluation to filter thousands of prompting papers. | |
| Categorizing Prompt Strategies and Deconstructing Role-Based Prompting | 7 | 5 | 4 | 6 | Sander takes a contrarian stance by declaring that role prompting and emotion prompting do not work for accuracy-based tasks on modern models, citing empirical benchmarks where an 'idiot' prompt beat a 'genius' prompt. Swyx pushes back vigorously, citing the Six Thinking Hats framework, Salesforce's DEI paper, and Tencent's billion-persona synthetic data generation to show where personas still provide reasoning value. | |
| Few-Shot Prompt Design Principles and Pitfalls | 7 | 5 | 1 | 3 | Sander reviews few-shot design parameters like exemplar ordering and in-distribution formatting. Swyx brings authentic production challenges regarding exemplar leakage in production newsletters, prompting Sander to explain literature showing that models prioritize output structure over exemplar label veracity. | |
| Thought Generation, Chain of Thought, and Decomposition Techniques | 7 | 4 | 3 | 6 | Sander shares his AutodiCoT technique and asserts that reasoning prompts like Chain of Thought should theoretically be obsolete due to post-training alignment. Swyx challenges this premise, noting that general-purpose foundation models inherently require explicit conditioning to focus attention on reasoning sub-spaces. | |
| Ensembling, Self-Consistency, and Cost-Aware Model Cascades | 8 | 3 | 3 | 7 | When discussing ensembling and cost, Sander suggests it is generally more practical to pay for top-tier frontier models rather than engineer complex routing cascades. Swyx pushes back strongly by presenting concrete token economics and his multi-tier production workflow where parallel calls to GPT-4o-mini are judged by a frontier model for significant net savings. | |
| Automated Prompt Engineering, DSPy, and the Role of AI Engineers | 7 | 2 | 0 | 1 | Sander admits how DSPy outperformed 20 hours of his manual prompt engineering in 10 minutes. Swyx connects this directly to his 'Rise of the AI Engineer' thesis, emphasizing that production prompting requires software engineering and programmatic optimization rather than standalone prompt craft. | |
| Jailbreaking, Prompt Injections, and Discoveries from Hack-A-Prompt | 7 | 6 | 2 | 2 | Alessio showcases his security background discussing CTFs and DEFCON red-teaming challenges. Sander provides a crisp breakdown differentiating prompt injection (conflicting developer vs user instructions) from jailbreaking (bypassing model safety filters directly), and highlights the context overflow attack discovered in Hack-A-Prompt. | |
| Multimodal Prompting Nuances, Structured Outputs, and Evaluation Pitfalls | 6 | 6 | 1 | 1 | After briefly exchanging notes on music and video generation difficulties, Sander provides a comprehensive critique of flawed LLM evaluation practices, specifically detailing why unanchored Likert scales and raw token log-probabilities yield misleading results in sensitive domains like healthcare. |
Statements from this episode (18)
Schulhoff: Preamble Discovered Prompt Injection Before Riley Goodside
“Preamble is the company that first discovered Prompt Injection, even before Riley, and they, like, responsibly disclosed it, kind of, internally to OpenAI”
Schulhoff: arXiv prohibits and removes undisclosed AI-generated papers
“I found AI-generated papers on Archive, and I flagged them to their staff, and they were like, thank you know, we missed these.
Wait, Archive takes them down?
Yeah.
Oh, I didn't know that.
You can't post an AI-generated paper there, especially
If you don't say…”
Schulhoff: Role Prompting Does Not Improve Accuracy on Modern LLMs
“For accuracy-based tasks, like MMLU, you're trying to solve a math problem, and maybe you tell the AI that it's a math professor, and you expect it to have improved performance. I really don't think that works. I'm quite certain that doesn't work on more moder…”
Schulhoff: Few-Shot Exemplar Order Can Shift Model Accuracy From 0% to 90%
“How you order your exemplars in the prompt is super important. And we've seen this move accuracy from like zero percent to 90%, like Zero to state of the art on some tasks, which is just ridiculous”
Schulhoff: LLMs Rely More on Prompt Structure Than Exemplar Labels
“There are a number of papers which have found that the label of the exemplar doesn't really matter, and the model reads the exemplars and cares more about structure than label.”
Swix: Few-Shot Prompting Is Not Always Superior to Zero-Shot Templates
“Few shot is not necessarily better than zero shot is, which is counterintuitive because you're working harder.”
Schulhoff: GPT-4 Fails to Output Reasoning on 1 in 100 to 1,000 Prompts
“I remember I did a lot of experiments with GPT-IV, and especially when you look at it at scale, so I'll run thousands of prompts against it through the API, and I'll see, you know, every one in a hundred, every one in a thousand outputs no reasoning whatsoever…”
Schulhoff: Self-consistency prompting yields diminishing returns on newer LLMs
“When it came out, it seemed to be quite performant, although more recently, I think as the models have improved, the
Performance of this technique has dropped, and you can see that in the evals we run near the end of the paper, where we use it, and it doesn't …”
Schulhoff: Researchers should pay for top models instead of engineering routing
“For the most part, designing these systems where you're kind of routing to different levels of intelligence is a really time-consuming and difficult task, and, like, it's probably worth it to just use the smart model
And pay for it at this point if you're look…”
Swix: Multi-sampling GPT-4o mini before GPT-4o judging yields net savings
“If I call a GP for a mini 10 times and I do a number of drafts or summaries, and then I have four, oh, judge the summaries that actually is net savings and like a good enough savings then running four, oh, on everything, which given the hundreds and thousands …”
Schulhoff: DSPy Beat 20 Hours of Manual Prompt Engineering in 10 Minutes
“And then I spent 20 hours prompt engineering for a task, and Dyspy beat me in 10 minutes, and that's when I changed my mind.”
Schulhoff: Automated Prompt Optimization Fails on Open Generation Without Ground Truth
“One limitation, I guess, is that you really need ground truth labels, so it's harder, if not impossible currently, to optimize open generation tasks, so like Writing, writing newsletters, I suppose. It's harder to automatically optimize those”
Schulhoff: Hiring Dedicated Prompt Engineers Makes No Sense for Most Companies
“I have always viewed prompt engineering as a skill that everybody should and will have, rather than a specialized role to hire for. That being said, there are definitely times where you do need just a prompt engineer. I think for AI companies, it's definitely …”
Schulhoff: Prompt injection overrides developer instructions; jailbreaking bypasses model directly
“Basically prompt injection is something that occurs when there is developer input, In the prompt, as well as user input in the prompt. So the developer instructions will say to do one thing, the user input will say to do something else. Jailbreaking is when it…”
Schulhoff: Open competitions uncover LLM exploits that paid staff never find
“What's really nice about competitions is that there is stuff that you'll just never find Paying people to do a job. And you'll only find it through random brilliant internet people inspired by thousands of people and the community around them all looking at th…”
Schulhoff: Prompting frameworks obscure hidden instructions and hurt reproducibility
“There's a lot of invisible prompts at work on a lot of these frameworks. I hate that. So like, you'll have Oh, this function summarizes input. But if you look behind the scenes, it's using some special summarization instruction. And if you don't have visibilit…”
Schulhoff: LLMs Have Number Biases and Require Explicit Rubrics for Evaluation
“These methods are super problematic because there is an incredible amount of instability in them, in the sense that models are biased towards outputting certain numbers, and you generally shouldn't say things like, output your result as a number on a scale of …”
Schulhoff: Hack-A-Prompt 2 Aims to Award $500,000 for Harmful AI Dataset
“We're looking to raise and then give away a half million dollars in prizes, and we're going to be creating the most harmful data set ever created, in the sense that this year we're going to be asking people to generate, force the models to generate real-world …”