The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 35 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Kolter: Neural network architectures matter less than commonly believed
“I actually think architectures don't matter as much as everyone else thinks they do.”
AI capabilities would have been achieved even without inventing transformers
“I think if we hadn't invented the transformer, we would have gotten there with whatever LSTM you know, state space model, whatever, anything else people were developing, we would have gotten there.”
Kolter: AI models do not get safer automatically by scaling up
“You can't just sort of trust models to get safer by getting bigger. You have to put in the work to actually make them safer. And this is, I think what a lot of AI companies are investing in. This is why we in fact do have models that are improving on these dim…”
Kolter: Robotics AI is not yet ready for pure compute scaling
“Certain fields. I think things like robotics is still one. I don't think we're quite at the, let's just scale it up level with robotics yet. Some companies might argue we are. I don't think we are. I think we're still in the let's explore methods to find the r…”
Kolter: Adversarial prompts optimized on open-source LLMs break commercial models
“Once we had done that, we found that when you had these weird terms that you sort of flipped around to optimize one, to optimize the response for one model, you could just take those same exact strings you would optimize, paste them into a commercial model, an…”
Kolter: AI agent benefits outweigh security risks if deployed with proper guardrails
“Yes, I think so, actually. I think if you run with proper guardrails, you know, we release guardrails for coding agents, for example. If you're on proper guardrails with proper sandboxing, and right now, yes, you probably also take some care to be a little bit…”
Kolter no longer writes code manually, relying entirely on AI agents
“I don't write code anymore. I do all my work now, and I do lots of, you know, I still do some research, right? It's entirely telling Codex what to do.”
The vast majority of modern AI intelligence comes from self-training
“I don't think people have properly internalized the fact that the vast majority of intelligence comes from self training effectively.”
Zico Kolter: Current AI trajectory will yield capable systems without breakthroughs
“I think the current trajectory we're on is going to get us, even if there were no more breakthroughs, I think, you know, with the minor additions that we are doing right now, we will get to incredibly capable systems, even if we were to freeze things right now…”
Text model scaling is one of humanity's top scientific discoveries
“The discovery That when you train big enough models on lots of text, and then turn, and then a little bit of additional sort of, you know, fine-tuning text, and then turn them loose to generate, that this generates long-form coherent thought. That was probably…”
OpenAI's Safety Committee has authority to delay model releases
“In the case essentially where we have more questions, we can delay model release if we feel that we need to understand that, that better.”
Zico Kolter: AI models are objectively safer than a year ago
“Models, definitely, objectively, I would say, in a lot of scenarios we can measure, are safer than they were a year ago.”
Kolter: Calculating a P(doom) for AI is a flawed concept
“I have never expressed a P. Doom and things like this. I just think it's a very weird concept as if the world is some stochastic set of dice that you can roll multiple times and that we don't have direct influence over this.”
Kolter: AI safety is solved through frontier interaction, not pauses
“I think the way you solve things is through, through ongoing exploration of what's happening and through, through interaction with the frontier.”
Kolter: Reasoning models are much harder to jailbreak via probability optimization
“Reasoning models were much more effective because you can't really do the same trick of optimizing for a probability with a reasoning model that has a whole trace of reasoning that happens in the middle and kind of reflect a bit more. So it's much harder to br…”
Kolter: AI coding agents are extremely good mechanistic interpretability researchers
“Coding agents are extremely good Mechinterp researchers.”
Kolter: AI agents might turn mechanistic interpretability into a science
“I think that we actually might finally be able to make more what I would consider a science of this through essentially leveraging mass research by agents deployed for this problem.”
Kolter: Scientific progress occurs when young researchers ignore old guard beliefs
“Basically progress happens when The current crop of young researchers ignores the things they've been taught that the old guard believes.”
A complete modern LLM takes only 200 to 300 lines of Python
“You have this code this code to build a complete large language model that can train on a large data set and learn to speak, runs on GPUs yes, eventually is trained with RL and tool calls. That entire set of code, probably two to 300 lines of Python code.”
Kolter: AI companies should establish board-level safety and security committees
“I think it's actually very important that AI companies start to establish similar governance policies, because this is something that requires that level of just oversight and of assurance. It is a becoming You know, a massive industry, and just like there are…”
Kolter: AI safety focus is shifting from single models to ecosystems
“I think actually one of the big trends we're seeing is a lot of safety is moving from the model level to the ecosystem level and talking about, you know, what's not one model capable of, but what's AI broadly capable of.”
Kolter: Renaming AI Safety Summit to AI Action Summit shows political shift
“The, you know, AI safety conference was, or AI safety summit was renamed the AI action summit or something is, has some significance actually in terms of the sort of taking temperature of where the world is politically.”
OpenAI differentiated early on by prioritizing model scale over new methods
“About opening up early on is that they always had this bet on scale. In a time where I think that was looked upon very suspiciously that, oh, if you, the thought somehow that we had all the methods already, and all you had to do was scale them up that mindset …”
Kolter: Universities will drive AI breakthroughs in math and science
“There's going to be a whole lot of breakthroughs happening with AI enablement in math and basic science, those kinds of things. Universities, I think will play a foundational role in shaping that future.”
Kolter: AI evals test average performance; security tests worst-case performance
“Most evaluations are done kind of in a, They measure expected value, basically. They measure sort of how well does it work on average, and security measures how well does it work in the worst case.”
Kolter: Jailbreaking modern AI safety systems requires complex multi-query attacks
“But they are, they require that degree of complexity to really jailbreak modern systems in a, for information that has this sensitivity to it.”
Kolter: Prompt injection introduces data exfiltration risks to AI agents
“Things like prompt injection are really a new security vulnerability for AI agents, and they mean that your risk is not just that you could have some, the model says something mean to you or something like that. Or even they could just write bad code. It could…”
Kolter: AI agent security requires managing permissions alongside manipulation risks
“AI security of agents is this interaction between what can the agent be manipulated into doing? What might it do accidentally? And what credentials or access does it have to really affect change?”
Zico Kolter: Reinforcement learning is now the foundation of all AI post-training
“RL is now the foundation of really all post training. It's all done by RL.”
Kolter: Major AI breakthroughs require both massive scale and luck
“Reasoning models were the next big breakthrough. Those are rare. They do take kind of a, you know, both, both a massive scale and kind of a bit of luck to get there.”
Kolter: Entire complexity of AI systems emerges from training data
“The entire complexity of an AI system evolves from the data they're trained on.”
Zico Kolter: Agentic AI systems have far more autonomy than last year
“The amount of autonomy granted to agentic systems now is far greater than a year ago.”
Kolter tried recruiting Schulman and Karpathy to CMU before OpenAI launched
“I was trying to get both John Schulman and Andre Karpathy to apply for faculty jobs at CMU, and I was trying to understand where they were, if they were going to apply, what they're going to be like, and they said, no, I think I'm going to be doing this startu…”
Kolter: State-of-the-art AI security combines classifiers, safety training, and opsec
“What they look like is basically classifiers on input. So you'll read what the, what a user types in classifiers on things like tool responses to classifiers. And when I say classifier, I just mean things that will read texts and kind of classify whether or no…”
CMU undergrad AI course has students build an LLM from scratch
“You build a LLM completely from scratch. You use PyTorch, but you build one from scratch that, you know, can be a chatbot. You train it on data. You RL it to solve math problems with tool calls. You do all of this. And this is a undergrad level course.”