Insight
Carlini: If LLMs always give desired answers, questions aren't hard enough
“When you're using these models, if you're getting the answer you want, always, it means you're not asking them hard enough questions.”
Insight
Carlini: 90% of scientific research is routine work that AI can automate
“90% of this is not doing something new. Like, 90% of this is like doing things a million people have done before, and then a little bit of something that was new. There's a reason why we say we stand on the shoulders of giants. It's true. Almost everything tha…”
Insight
Carlini: Users should build personalized AI benchmarks instead of relying on public leaderboards
“The argument that I tried to lay out in this post is that more people should make benchmarks that are tailored to them.”
Insight
Carlini: GPT-4 would exist identically without adversarial machine learning research
“Nothing about GPT-IV would be at all different if the field of, like the entire field of Everson machine learning disappeared. Like everything to do with Everson examples, like all of the, like for the most part, like GPT-IV would exist identically.”
Opinion
Carlini: Most AI commentators spin arguments based on ideology rather than reality
“I feel like most people who write about language models being good or bad, some underlying message of like, you know, they have their camp and their camp is like, AI is bad or AI is good or whatever. And they like, they spin whatever they're gonna say accordin…”
Insight
Carlini: Copying and pasting error messages effectively creates a coding agent
“Currently though, make a model into an agent by just copying and pasting error messages for the most part. And that's what I do is, you know, you run it and it gives you some code that doesn't work and either I'll fix the code or it will give me buggy code and…”
Insight
Carlini: Language models should not be trusted in adversarial situations
“My research says is entirely on this. Like you probably shouldn't trust these models to do the things in adversarial situations.”
Insight
Carlini: Always qualify claims about AI with 'for current models'
“Whenever someone says X is true about language models, you should always append the suffix for current models, because I'll be the first to admit I was one of the people who was very much on the opinion that these language models are fun toys and are going to …”
Insight
Carlini: Unpopular AI benchmarks protect against model contamination and overfitting
“And by having a benchmark that is not very popular, you can be relatively certain that no one has tried to optimize their model for your benchmark.”
Insight
Carlini: LLM-as-a-judge is almost always accurate when prompted correctly
“I've inspected the outputs of these and like, they're almost always correct. If you sort of, if you ask the model to judge these things in the right way, they're very good at being able to tell this.”
Opinion
Carlini: ML security research failed to adapt to real-world systems
“And then machine learning started to work. And the thing that bothered me is it seems like the other machine learning community didn't then try and adapt and try and actually start studying real problems.”
Assertion Supported
Attackers can poison the LAION dataset simply by purchasing expired domains
“Here's this new dataset. It is being distributed in such a way that anyone in the world can buy domains that let you then inject arbitrary images in the dataset.”
Assertion Supported
Carlini extracted production models from Google and OpenAI with legal permission
“We ran the attack that let us, yeah, stole several of OpenAI's models. With their permission... We notified everyone who was vulnerable to this attack. Some Google models were vulnerable. Some open AM models were vulnerable. There were one or two other people …”
Assertion Supported
Prompting ChatGPT to repeat a word indefinitely leaks verbatim training data
“One of my co-authors, Milad was working on some other random experiments, and he figured out that if you prompt ChatGPT to repeat a word forever, then it will repeat the word many, many, many times in a row, and then like explode and like just start doing rand…”
Assertion Supported
Carlini: ChatGPT emitted verbatim 50+ word sequences from internet training data
“And what I can say is that the output of the model was a verbatim, at least 50 word in a row match. To some other document that appeared on the internet previously.”
Insight
Carlini: Imperfect LLMs remain useful because users already distrust internet content
“You can't trust these things blindly, but I feel like most people on the internet already understand that things on the internet you can't trust blindly. And so there's not like, this is not like a big mental shift you have to go through to understand that it …”
Insight
Carlini: LLMs reduce onboarding to unfamiliar tools from hours to 10 minutes
“It would have taken me. You know, several hours to figure out some things that take 10 minutes if you could just ask exactly the question you want the answer to.”
Insight
Carlini: AI helper functions preserve programmer mental state on complex problems
“One of the ways we currently don't think about being distracted is you're solving some hard problem and you realize you need a helper function that does X where X is like, it's a known algorithm... Instead of using my mental capacity and solving that problem, …”
Assertion Not checkable as stated
Carlini: LLMs decompile obscure binaries into readable Python code
“It can turn the compiled source code, which is impossible for any human to understand into the Python code that is entirely reasonable to understand. And, you know, it doesn't run. It has a bunch of problems, but like, it's so much nicer that it's immediately …”
Insight
Carlini: Anyone claiming 0% or 100% certainty on 5-year AI capabilities is probably wrong
“If you would say there's a zero percent chance that something, you know, the models will get very, very good in the next five years, you're probably wrong. If you're going to say there's a hundred percent chance that in the next five years, some, then you're p…”
Insight
Carlini: AI security research depends on whether models act autonomously or keep humans in loop
“The way in which security intersects with these things depends a lot in exactly how people use these tools. You know, if it turns out to be the case that these models get to be truly amazing and can solve, you know, tasks completely autonomously, that's a very…”
Insight
Carlini: If prompt engineering takes longer than manual work, LLMs save no time
“If I have to spend so much time thinking about how I want to frame the question that it would have been faster for me just to get the answer. Didn't save me any time. And so oftentimes, you know, what I do is like, I just dump in whatever current thought that …”
Disclosure
Carlini never uses few-shot prompting for personal language model queries
“I don't because usually when I want the answer, I just, I want to get the answer.”
Opinion
Carlini: AI community should do more multi-turn evaluations
“This is the thing that I think many people should be doing more of. I would like more multi-turn evals. I might be writing a paper on this at some point if I get around to it.”
