Opinion
Schulhoff: Role Prompting Does Not Improve Accuracy on Modern LLMs
“For accuracy-based tasks, like MMLU, you're trying to solve a math problem, and maybe you tell the AI that it's a math professor, and you expect it to have improved performance. I really don't think that works. I'm quite certain that doesn't work on more moder…”
Assertion Supported
Schulhoff: LLMs Rely More on Prompt Structure Than Exemplar Labels
“There are a number of papers which have found that the label of the exemplar doesn't really matter, and the model reads the exemplars and cares more about structure than label.”
Assertion Not checkable as stated
Schulhoff: DSPy Beat 20 Hours of Manual Prompt Engineering in 10 Minutes
“And then I spent 20 hours prompt engineering for a task, and Dyspy beat me in 10 minutes, and that's when I changed my mind.”
Opinion
Schulhoff: Hiring Dedicated Prompt Engineers Makes No Sense for Most Companies
“I have always viewed prompt engineering as a skill that everybody should and will have, rather than a specialized role to hire for. That being said, there are definitely times where you do need just a prompt engineer. I think for AI companies, it's definitely …”
Insight
Schulhoff: Prompt injection overrides developer instructions; jailbreaking bypasses model directly
“Basically prompt injection is something that occurs when there is developer input, In the prompt, as well as user input in the prompt. So the developer instructions will say to do one thing, the user input will say to do something else. Jailbreaking is when it…”
Assertion Supported
Schulhoff: Preamble Discovered Prompt Injection Before Riley Goodside
“Preamble is the company that first discovered Prompt Injection, even before Riley, and they, like, responsibly disclosed it, kind of, internally to OpenAI”
Assertion Supported
Schulhoff: Few-Shot Exemplar Order Can Shift Model Accuracy From 0% to 90%
“How you order your exemplars in the prompt is super important. And we've seen this move accuracy from like zero percent to 90%, like Zero to state of the art on some tasks, which is just ridiculous”
Assertion Not checkable as stated
Schulhoff: GPT-4 Fails to Output Reasoning on 1 in 100 to 1,000 Prompts
“I remember I did a lot of experiments with GPT-IV, and especially when you look at it at scale, so I'll run thousands of prompts against it through the API, and I'll see, you know, every one in a hundred, every one in a thousand outputs no reasoning whatsoever…”
Insight
Schulhoff: Researchers should pay for top models instead of engineering routing
“For the most part, designing these systems where you're kind of routing to different levels of intelligence is a really time-consuming and difficult task, and, like, it's probably worth it to just use the smart model
And pay for it at this point if you're look…”
Insight
Schulhoff: Open competitions uncover LLM exploits that paid staff never find
“What's really nice about competitions is that there is stuff that you'll just never find Paying people to do a job. And you'll only find it through random brilliant internet people inspired by thousands of people and the community around them all looking at th…”
Insight
Schulhoff: Prompting frameworks obscure hidden instructions and hurt reproducibility
“There's a lot of invisible prompts at work on a lot of these frameworks. I hate that. So like, you'll have Oh, this function summarizes input. But if you look behind the scenes, it's using some special summarization instruction. And if you don't have visibilit…”
Insight
Schulhoff: LLMs Have Number Biases and Require Explicit Rubrics for Evaluation
“These methods are super problematic because there is an incredible amount of instability in them, in the sense that models are biased towards outputting certain numbers, and you generally shouldn't say things like, output your result as a number on a scale of …”
Assertion Supported
Schulhoff: arXiv prohibits and removes undisclosed AI-generated papers
“I found AI-generated papers on Archive, and I flagged them to their staff, and they were like, thank you know, we missed these.
Wait, Archive takes them down?
Yeah.
Oh, I didn't know that.
You can't post an AI-generated paper there, especially
If you don't say…”
Assertion Supported
Schulhoff: Self-consistency prompting yields diminishing returns on newer LLMs
“When it came out, it seemed to be quite performant, although more recently, I think as the models have improved, the
Performance of this technique has dropped, and you can see that in the evals we run near the end of the paper, where we use it, and it doesn't …”
Insight
Schulhoff: Automated Prompt Optimization Fails on Open Generation Without Ground Truth
“One limitation, I guess, is that you really need ground truth labels, so it's harder, if not impossible currently, to optimize open generation tasks, so like Writing, writing newsletters, I suppose. It's harder to automatically optimize those”
Disclosure
Schulhoff: Hack-A-Prompt 2 Aims to Award $500,000 for Harmful AI Dataset
“We're looking to raise and then give away a half million dollars in prizes, and we're going to be creating the most harmful data set ever created, in the sense that this year we're going to be asking people to generate, force the models to generate real-world …”
