Ameisen: Interpretability research has lower entry barriers and low compute needs
Emmanuel Ameisen · The Utility of Interpretability — Emmanuel Amiesen · Jun 6, 2025 · at 28:56
Anthropic research scientist Emmanuel Ameisen explains why mechanistic interpretability is uniquely accessible to software engineers transitioning into AI research.
0:00 / 0:49exact quote · 49.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think for Interp in particular, there's like another thing that makes it easier to transition to, which is maybe two things. One, you can just do it without huge access to compute. Like, there are open source models. You can look at them. A lot of Interp papers, you know, coming out of programs like Maths are on models that are open source that you can sort of like dissect without having a cluster of like, you know, a hundred GPUs. You can just even Sometimes you can load them, like, on your CPU, on your MacBook, and it's also a relatively new field, and so, you know, there's, as I'm sure we'll talk about, there's, like, some conceptual burdens and concepts that you just want to, like, understand before you contribute, but it's not You know, physics. It's relatively recent, and so the number of abstractions that you have to, like, ramp up on is just not that high compared to other fields, which I think makes that transition somewhat easier.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Emmanuel Ameisen
Ameisen: Stochastic Parrots Label Ignores Complex Multi-Step LLM Reasoning
“It's, like, activating many different distributed representations, like, combining them, and sort of, like, doing something pretty complicated. And so, yeah, I think it's funny, because in my opinion, that's like, yeah, like, oh god, stochastic parrots is not …”
Ameisen: LLMs plan future tokens rather than operating purely myopically
“Language models are next token predictors is like a fact. Like that is what they do. They are trained to predict the next token. However, that does not mean that they myopically only consider the next token When they choose the next token, you can work on brea…”
Ameisen: Deceptive Backward Reasoning Exists in Base Pre-Trained Models
“I bet, I don't know how much I bet a hundred bucks. So somebody can like, they would get a hundred bucks from me if they prove that I'm wrong, that this behavior for a model that does a drink fine tuning, it also does it post pre-training.”
Ameisen: Current LLM Chain of Thought Is Unfaithful and Untrustworthy
“So I think there's like a sense in which right now the chain of thought is, is unfaithful, or at least you can't read the chain of thought and trust that that's how the model did it.”
Ameisen: Multi-Hop Reasoning Circuits Are Extremely Similar Across Small and Large Models
“The way the circuit looks in Gemma, like a really small model is extremely similar to the way that it looks like a huge model, which that in itself is, I think like a pretty novel discovery. It's like, oh, you have these models that are like super different. Y…”
Ameisen: Anthropic's circuit tracing tool ignores attention heads and only decomposes MLPs
“These are just MLPs. So the model has both attention heads and multi-layer perceptions MLPs. We don't just do it. Like we completely ignore attention or like we don't try to decompose it at all. So there's some prompts where like all of the interesting stuff i…”