Aug 28, 2024 · 1h 7m · latent-space
Personal benchmarks vs HumanEval - with Nicholas Carlini of DeepMind
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
DeepMind research scientist Nicholas Carlini joins the Latent Space podcast to discuss pragmatic LLM workflows for developers, the necessity of personalized AI benchmarks over public leaderboards, and his groundbreaking adversarial security research.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 4.6 Guest teaching 5.3 Guest disagreement 2.3 The hosts pushing back 2.6
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:02:54 Carlini firmly refutes skepticism regarding training data memorization
When Swix questions how LLMs can memorize training data given parameter limits, Carlini forcefully counters with concrete empirical evidence of 50-to-100+ word verbatim text extractions from the internet.
Hardest push from the hosts ▶ 1:03:56 Alessio confronts Carlini with his boss's challenge on defense workAlessio directly channels DeepMind CISO Vijay to challenge Carlini on why he exclusively focuses on breaking systems rather than constructing defenses.
Biggest teaching moment ▶ 3:42 Carlini schools Swix on Turing completeness in printfWhen Swix assumes printf cannot have control flow, Carlini educates him on how the %n specifier and loop indexing enable loops, conditionals, and Turing complete execution in C.
The host holds their own ▶ 50:57 Swix devises benchmark canary tests that match cutting-edge dataset inference researchSwix formulates an adversarial canary evaluation method to catch overtraining on benchmarks, which Carlini validates by connecting it directly to Stanford dataset inference papers.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Turing Completeness in Game of Life and Printf | 4 | 7 | 3 | 4 | Alessio and Swix ask about Carlini's obsession with Turing completeness in Conway's Game of Life. When Swix challenges whether printf can handle control flow, Carlini educates him in detail on how the %n specifier allows arbitrary memory writes, loops, and conditionals. | |
| Motivations Behind the 'How I Use AI' Blog Post | 3 | 3 | 2 | 1 | The hosts ask Carlini about his viral blog post on AI usage. Carlini explains his neutral motivation to ground practical reality between extreme AI hype and complete dismissal. | |
| Building Applications and Practical Agentic Workflows with LLMs | 4 | 4 | 1 | 2 | Alessio asks about building web apps and where ChatGPT stands relative to full agents. Carlini explains how he treats simple copy-paste error loops as an effective agent workflow for frontend boilerplate. | |
| Accelerating Unfamiliar Technologies and Grounded Use Cases | 4 | 4 | 1 | 2 | Carlini discusses using LLMs to learn targeted tools like Docker without studying first principles. Alessio asks about knowledge cutoff limitations, and Carlini clarifies his scope as an independent researcher on small projects. | |
| Ephemeral Software and Pushing LLM Capability Limits | 5 | 4 | 2 | 2 | Swix and Alessio discuss ephemeral software, with Alessio sharing a practical example of repairing an irrigation system. Carlini notes that if an LLM always gives you the right answer, you are not asking hard enough questions. | |
| Underrated LLM Workflows: Debugging and Offloading Mundane Tasks | 3 | 6 | 2 | 1 | Carlini shares underrated workflows, specifically having LLMs write routine helper algorithms to avoid breaking mental flow. He refutes the claim that LLMs cannot help research by explaining that 90% of cutting-edge research relies on established methods. | |
| Expert vs. Non-Expert Adoption and Trusting AI Systems | 5 | 6 | 3 | 3 | Swix asks about expert versus non-expert gains and questions whether papers showing LLMs produce insecure code have an agenda. Carlini defends Dan Boneh's research while admitting his own past mistake in writing off early LLMs and RLHF. | |
| Code Analysis, API Navigation, and Binary Decompilation | 4 | 5 | 1 | 1 | Alessio prompts best practices for learning and code analysis. Carlini details using LLMs for cryptic CLI flags like FFmpeg and decompiling reverse-engineered binary assemblies into readable Python. | |
| Nuanced AI Forecasting and Authentic Online Distribution | 4 | 4 | 2 | 2 | Swix inquires about Carlini's upcoming writing and forecasting. Alessio asks about his unconventional distribution strategy, and Carlini explains maintaining only an RSS feed and email list to avoid toxic social media arguments. | |
| Non-AI Passion Projects: Paper Data Storage and IOCCC | 5 | 6 | 2 | 3 | Carlini quizzes the hosts on data density per printed page (achieving 1.44MB) and explains why standardized benchmarks fail individual needs. Swix notes that publishing a personal benchmark risks contamination, prompting Carlini to clarify his goal was inspiring others to build bespoke evals. | |
| The Limits of Prompt Engineering and Lazy Prompting | 5 | 5 | 4 | 3 | Alessio brings up prompt engineering and Carlini's lazy prompting stance. Carlini rejects being told he is 'holding it wrong' and argues that needing elaborate prompt incantations defeats the time-saving purpose of using LLMs. | |
| Multi-Turn Evals and Detecting Benchmark Contamination | 6 | 6 | 2 | 4 | Swix highlights the limitation of single-turn evals in chat-centric workflows. Swix proposes inserting deliberate adversarial errors to detect benchmark memorization, and Carlini connects this to formal dataset inference research. | |
| Practical Security Research and LAION-400M Dataset Poisoning | 6 | 5 | 2 | 2 | Alessio brings up Carlini's security background and the LAION-400M poisoning attack. Carlini explains the transition from unrealistic academic ML threat models to practical real-world vulnerabilities like buying expired dataset domains. | |
| Model Stealing Attacks on Production LLM APIs | 6 | 6 | 3 | 3 | Alessio and Carlini discuss extracting projection layers from production APIs like OpenAI. Carlini details coordinating with corporate lawyers to steal layers and forcing API providers to patch logit bias disclosures. | |
| Training Data Extraction and Memorization in LLMs | 5 | 8 | 4 | 4 | Carlini mentions eliciting training data by prompting ChatGPT to repeat a single word indefinitely. When Swix pushes back that models lack parameter capacity to memorize training data, Carlini decisively proves memorization by citing 50-to-100+ word verbatim document extractions. | |
| Why Nicholas Carlini Attacks: Passion and Impact | 5 | 5 | 3 | 4 | Alessio delivers a direct question from DeepMind CISO Vijay asking why Carlini refuses to build defenses. Carlini defends attacking over defending based on genuine passion, asserting that forcing people into unfulfilling work leads to lower overall output. |
Statements from this episode (31)
Carlini published a paper proving C's printf function is Turing-complete
“A while ago as part of a research paper, I was able to show that in C, if you call into print def. It's Turing complete, like printf, you know, like which, like, you know, you can print numbers or whatever, right?”
Carlini: Most AI commentators spin arguments based on ideology rather than reality
“I feel like most people who write about language models being good or bad, some underlying message of like, you know, they have their camp and their camp is like, AI is bad or AI is good or whatever. And they like, they spin whatever they're gonna say accordin…”
Carlini: Copying and pasting error messages effectively creates a coding agent
“Currently though, make a model into an agent by just copying and pasting error messages for the most part. And that's what I do is, you know, you run it and it gives you some code that doesn't work and either I'll fix the code or it will give me buggy code and…”
Carlini: Imperfect LLMs remain useful because users already distrust internet content
“You can't trust these things blindly, but I feel like most people on the internet already understand that things on the internet you can't trust blindly. And so there's not like, this is not like a big mental shift you have to go through to understand that it …”
Carlini: LLMs reduce onboarding to unfamiliar tools from hours to 10 minutes
“It would have taken me. You know, several hours to figure out some things that take 10 minutes if you could just ask exactly the question you want the answer to.”
Carlini: If LLMs always give desired answers, questions aren't hard enough
“When you're using these models, if you're getting the answer you want, always, it means you're not asking them hard enough questions.”
Carlini: 90% of scientific research is routine work that AI can automate
“90% of this is not doing something new. Like, 90% of this is like doing things a million people have done before, and then a little bit of something that was new. There's a reason why we say we stand on the shoulders of giants. It's true. Almost everything tha…”
Carlini: AI helper functions preserve programmer mental state on complex problems
“One of the ways we currently don't think about being distracted is you're solving some hard problem and you realize you need a helper function that does X where X is like, it's a known algorithm... Instead of using my mental capacity and solving that problem, …”
Carlini: Language models should not be trusted in adversarial situations
“My research says is entirely on this. Like you probably shouldn't trust these models to do the things in adversarial situations.”
Carlini: Always qualify claims about AI with 'for current models'
“Whenever someone says X is true about language models, you should always append the suffix for current models, because I'll be the first to admit I was one of the people who was very much on the opinion that these language models are fun toys and are going to …”
Carlini: LLMs decompile obscure binaries into readable Python code
“It can turn the compiled source code, which is impossible for any human to understand into the Python code that is entirely reasonable to understand. And, you know, it doesn't run. It has a bunch of problems, but like, it's so much nicer that it's immediately …”
Carlini: Anyone claiming 0% or 100% certainty on 5-year AI capabilities is probably wrong
“If you would say there's a zero percent chance that something, you know, the models will get very, very good in the next five years, you're probably wrong. If you're going to say there's a hundred percent chance that in the next five years, some, then you're p…”
Carlini: AI security research depends on whether models act autonomously or keep humans in loop
“The way in which security intersects with these things depends a lot in exactly how people use these tools. You know, if it turns out to be the case that these models get to be truly amazing and can solve, you know, tasks completely autonomously, that's a very…”
Carlini encodes 1.44 megabytes of data onto a single sheet of paper
“Yeah, okay. So it's about, in particular, it's about 1.44 megabytes.”
Carlini built a gate-level CPU emulation for the IOCCC
“I have A very fun gate level emulation of an old CPU that runs, like, fully precisely, and it's a fun kind of thing.”
Carlini: Users should build personalized AI benchmarks instead of relying on public leaderboards
“The argument that I tried to lay out in this post is that more people should make benchmarks that are tailored to them.”
Carlini: Wrote a domain-specific language for creating custom AI benchmarks
“So what I did is I wrote a domain specific language that you can, anyone can write for and say they can, you can take tasks that you have wanted models to solve for you and you can Put them into your benchmark.”
Carlini: Unpopular AI benchmarks protect against model contamination and overfitting
“And by having a benchmark that is not very popular, you can be relatively certain that no one has tried to optimize their model for your benchmark.”
Carlini: LLM-as-a-judge is almost always accurate when prompted correctly
“I've inspected the outputs of these and like, they're almost always correct. If you sort of, if you ask the model to judge these things in the right way, they're very good at being able to tell this.”
Carlini: If prompt engineering takes longer than manual work, LLMs save no time
“If I have to spend so much time thinking about how I want to frame the question that it would have been faster for me just to get the answer. Didn't save me any time. And so oftentimes, you know, what I do is like, I just dump in whatever current thought that …”
Carlini never uses few-shot prompting for personal language model queries
“I don't because usually when I want the answer, I just, I want to get the answer.”
Carlini: AI community should do more multi-turn evaluations
“This is the thing that I think many people should be doing more of. I would like more multi-turn evals. I might be writing a paper on this at some point if I get around to it.”
Carlini runs Docker-based multi-turn evals for Git cherry-picking tasks
“I spin up a Docker container. I run the con whatever the model told me the output to do is. I feed the output back into the model. I repeat this many rounds, and I check at the very end, you know, does the git commit history show that it is correctly cherry pi…”
Stanford researchers detect benchmark contamination by testing evaluation question ordering
“There's a paper by Tatsu at a Stanford. Where they check if the order that the specific questions happen to be in matters. And if the answer is yes, then you probably trained on it because the order of the questions is arbitrary and shouldn't matter.”
Carlini: Every image in LAION-400M is pulled from live domains
“Every image gets pulled from a live domain.”
Carlini: ML security research failed to adapt to real-world systems
“And then machine learning started to work. And the thing that bothered me is it seems like the other machine learning community didn't then try and adapt and try and actually start studying real problems.”
Attackers can poison the LAION dataset simply by purchasing expired domains
“Here's this new dataset. It is being distributed in such a way that anyone in the world can buy domains that let you then inject arbitrary images in the dataset.”
Carlini extracted production models from Google and OpenAI with legal permission
“We ran the attack that let us, yeah, stole several of OpenAI's models. With their permission... We notified everyone who was vulnerable to this attack. Some Google models were vulnerable. Some open AM models were vulnerable. There were one or two other people …”
Carlini: GPT-4 would exist identically without adversarial machine learning research
“Nothing about GPT-IV would be at all different if the field of, like the entire field of Everson machine learning disappeared. Like everything to do with Everson examples, like all of the, like for the most part, like GPT-IV would exist identically.”
Prompting ChatGPT to repeat a word indefinitely leaks verbatim training data
“One of my co-authors, Milad was working on some other random experiments, and he figured out that if you prompt ChatGPT to repeat a word forever, then it will repeat the word many, many, many times in a row, and then like explode and like just start doing rand…”