Ameisen: Superposition is more severe in language models than in vision models
Emmanuel Ameisen · The Utility of Interpretability — Emmanuel Amiesen · Jun 6, 2025 · at 35:01
Anthropic research scientist Emmanuel Ameisen explains the intuition behind the superposition hypothesis in LLMs.
0:00 / 1:00exact quote · 60.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“That means that like language models pack a lot more in less space than Vision models. So maybe like a kind of like really hand wavy analogy, right? It's like, well, if you want curve detectors, like you don't need that many curve detectors. You know, if each curve detector is going to detect like a quarter or a 12th of a circle, like, okay, well you have all your curve detectors, but think about all of the concepts that like Claude or even GPT-II need to know, like just in terms of, it needs to know about like all of the different Colors, all the different hours of every day, all of the different cities in the world, all of the different streets on every city. If you just enumerate all of the facts that like a model knows, you're going to get like a very, very long list. And that list is going to be way bigger than like the number of neurons or even the size of the residual stream, which is where like the models process information. And so there's this sense in which like, oh, there's more information than there's like dimensions to represent it. And that is much more true for language models than for vision models.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Emmanuel Ameisen
Ameisen: Stochastic Parrots Label Ignores Complex Multi-Step LLM Reasoning
“It's, like, activating many different distributed representations, like, combining them, and sort of, like, doing something pretty complicated. And so, yeah, I think it's funny, because in my opinion, that's like, yeah, like, oh god, stochastic parrots is not …”
Ameisen: LLMs plan future tokens rather than operating purely myopically
“Language models are next token predictors is like a fact. Like that is what they do. They are trained to predict the next token. However, that does not mean that they myopically only consider the next token When they choose the next token, you can work on brea…”
Ameisen: Deceptive Backward Reasoning Exists in Base Pre-Trained Models
“I bet, I don't know how much I bet a hundred bucks. So somebody can like, they would get a hundred bucks from me if they prove that I'm wrong, that this behavior for a model that does a drink fine tuning, it also does it post pre-training.”
Ameisen: Current LLM Chain of Thought Is Unfaithful and Untrustworthy
“So I think there's like a sense in which right now the chain of thought is, is unfaithful, or at least you can't read the chain of thought and trust that that's how the model did it.”
Ameisen: Multi-Hop Reasoning Circuits Are Extremely Similar Across Small and Large Models
“The way the circuit looks in Gemma, like a really small model is extremely similar to the way that it looks like a huge model, which that in itself is, I think like a pretty novel discovery. It's like, oh, you have these models that are like super different. Y…”
Ameisen: Anthropic's circuit tracing tool ignores attention heads and only decomposes MLPs
“These are just MLPs. So the model has both attention heads and multi-layer perceptions MLPs. We don't just do it. Like we completely ignore attention or like we don't try to decompose it at all. So there's some prompts where like all of the interesting stuff i…”