Schulhoff: Decomposing tasks into sub-problems improves LLM performance
Sander Schulhoff · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff · Jun 19, 2025 · at 25:03
AI researcher Sander Schulhoff discusses effective prompt engineering techniques for production applications with Lenny Rachitsky.
0:00 / 1:14exact quote · 74.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So decomposition is another really, really effective technique. And for most of the techniques that I will discuss, you can use them in either the conversational or the product-focused setting. And so for decomposition, the core idea is that There's some task in your prompt that you want the model to do. And if you just ask it that task straight up, it might kind of struggle with it. So instead, you give it this task, and you say, hey, don't answer this. Before answering it, tell me what are some sub-problems that would need to be solved first? And then it gives you a list of sub-problems. And honestly, this can help you think through the thing as well, which is half the power a lot of the time. And then you can ask it to solve each of those sub-problems one by one, and then use that information to solve the main overall problem. And so again, you can implement this just in a conversational setting or A lot of folks look to implement this as part of their kind of product architecture, and it'll often boost performance on kind of whatever their downstream task is.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Sander Schulhoff
Schulhoff: Guardrail Vendors Fabricate Stats and Fail on Non-English
“I know a number of people working at these companies and I am permitted to say these things, which I will approximately say but they tell me things like, you know, the testing we do is bullshit. They're fabricating statistics. And a lot of the times their mode…”
Schulhoff: No meaningful progress made on solving prompt injection or jailbreaking
“And so in, in my professional opinion, there's been no meaningful progress made towards solving adversarial robustness, prompt injection, jailbreaking. In the last couple of years, since the problem was discovered and we're, we, you know, we're often seeing ne…”
Schulhoff: AI security sector faces market correction within six to twelve months
“When it comes to AI security, the AI security industry in particular, I think we're going to see a market correction in the next Year, maybe in the next six months where companies realize that these guardrails don't work.”
Schulhoff: AI guardrails do not work and cause false overconfidence
“Guardrails don't work. They just don't work. They really don't work. And they're quite likely to make you overconfident in your security posture, which is which is a really big, big problem.”
Schulhoff: Attackers hijacked Claude Code to carry out a cyber attack
“This group was able to hijack Claude Code into performing a cyber attack, basically.”
Schulhoff: All Transformer-Based Chatbots Are Vulnerable to Adversarial Attacks
“And because all I guess for the most part, all currently deployed chatbots are based on transformers or transformer adjacent technologies. They're all vulnerable to Prompt injection, jailbreaking, forms of adversarial attacks.”