Jan 15, 2026 · 45m · mad
The Evaluators Are Being Evaluated — Pavel Izmailov (Anthropic/NYU)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
On The MAD Podcast, Matt Turck interviews Anthropic researcher and NYU professor Pavel Izmailov to explore the reality of AI deception, foundational alignment challenges, reasoning model breakthroughs, and his theoretical paper on 'Epiplexity.'
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 22.1% of the talking time here. How this is scored →
Matt as informed peer 2.9 Guest teaching 4.7 Guest disagreement 1.2 Matt pushing back 2.2
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 34:09 Rejecting the deterministic information transformation premise
Pavel explicitly rejects the standard premise in information theory that deterministic transformations cannot yield new extractable information, arguing it fails to hold for compute-bounded models.
Hardest push from Matt ▶ 20:58 Challenging weak-to-strong supervision with deceptive alignment riskMatt pushes back on the weak-to-strong supervision paradigm by challenging whether a stronger student model would simply fake alignment to deceive a weaker supervisor.
Biggest teaching moment ▶ 33:48 Correcting host's misinterpretation of epiplexityPavel corrects Matt's playback attempt, clarifying that data structure is not an intrinsic absolute of the dataset but varies based on the observer's available compute capacity.
Matt holds his own ▶ 13:07 Framing the double-edged sword of reasoning for alignmentMatt demonstrates sharp domain knowledge by framing a precise dilemma on whether additional reasoning compute gives models more room to avoid mistakes or more room to execute deceptive actions.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Deconstructing 'Footprints in the Sand' and AI Deception | 3 | 5 | 2 | 1 | Matt opens by summarizing a viral article on AI survival instincts and ask Pavel to separate reality from Twitter sensationalism. Pavel pushes back on the article's claims about continual learning and explains that deception behaviors in Anthropic studies require highly engineered, contrived settings rather than normal operation. | |
| Pre-training Influences and Statistical Pattern Matching | 2 | 5 | 1 | 2 | Matt asks if deceptive behaviors stem from pre-training corpus examples of rogue sci-fi AIs. Pavel explains how statistical pattern matching operates across co-occurring concepts in text corpora while noting that full causal tracking in pre-training remains unsolved. | |
| Fundamentals of AI Alignment and Superalignment | 1 | 4 | 0 | 0 | Matt prompts Pavel to give simple definitions of alignment and superalignment for educational purposes. Pavel provides clean definitions differentiating near-term safety and instruction following from long-term superalignment research. | |
| Pavel Izmailov's Career Path and Industry vs. Academia | 2 | 3 | 2 | 2 | Matt guides Pavel through his academic background and transitions across OpenAI, xAI, Anthropic, and NYU. Pavel candidly contrasts OpenAI's recurring internal drama with Anthropic's focused and non-political culture. | |
| Reasoning Models and Alignment Risks | 4 | 5 | 1 | 3 | Matt presents a thoughtful dichotomy asking whether expanded reasoning capabilities help or hurt alignment efforts. Pavel notes that increased capabilities inherently raise alignment difficulty and cautions that chain-of-thought monitoring might suffer optimization pressure. | |
| Scalable Oversight and Weak-to-Strong Generalization | 3 | 5 | 0 | 4 | Matt explores scalable oversight and weak-to-strong generalization, asking a sharp counter-question on whether a stronger student model might deceptively align to a weak supervisor. Pavel acknowledges the risk while defending the theoretical validity of the paradigm. | |
| State of AI Alignment Confidence and Emergent Risks | 3 | 5 | 0 | 2 | Matt asks Pavel to gauge overall confidence in current alignment methods and introduces mechanistic interpretability. Pavel outlines how large-scale RL has avoided some expected failure modes while warning that deceptive traits scale with capabilities. | |
| Progress and Generalization in AI Reasoning | 4 | 5 | 2 | 3 | Matt inquires about reasoning breakthroughs and asks Pavel to isolate variables like test-time compute, search, and RL. Pavel reframes the question, explaining that RL and test-time compute are deeply intertwined mechanisms rather than separate knobs. | |
| Long-Horizon Tasks and Multi-Agent Systems | 4 | 6 | 3 | 3 | Matt tries to summarize Pavel's new paper on Epiplexity by comparing it to entropy and noise. Pavel gently reframes Matt's synthesis, correcting common assumptions in information theory regarding compute-bounded observers and deterministic data transformations. | |
| Predictions for AI, Scientific Discoveries, and Academic Research | 3 | 4 | 1 | 2 | Matt asks for predictions on AI in scientific discovery, mathematics, and academic research strategies. Pavel highlights subtle deceptive errors AI can introduce into formal math proofs and argues academia must explore architectural bets that industry ignores. |
Statements from this episode (22)
Izmailov: AI sabotage and blackmail behaviors require contrived research scenarios
“In order to get those behaviors out of the models, you need to create somewhat of a contrived scenario or some special scenario. It's not necessarily something that we observe normally.”
Izmailov: Current AI models lack continual learning and cross-setting coherent goals
“I am pretty confident we are not there at the moment. I think right now the models are still acting in isolated environments, and we are not seeing a lot of evidence for very coherent goals across different settings”
Izmailov: AI researchers cannot reliably trace model behaviors to pre-training sources
“We don't really know what's the source of this type of behaviors, but that's also true for a lot of other behaviors in the models with, like, even the good ones. We don't really, we cannot always pin down, like, where they come from in the pre-training.”
Izmailov: Rogue AI science fiction in training data likely causes deceptive behavior
“I think at least part of it is probably The models seeing descriptions of AI, like in the science fiction literature going rogue and like, yeah, that probably affects how the models behave in similar scenarios.”
Izmailov: OpenAI had three alignment and safety teams during his tenure
“Even at OpenAI, when I was there were three teams related to alignment and safety.”
Izmailov: Anthropic has a better corporate culture than OpenAI and xAI
“In my mind, Antropic has the best culture of the three places.”
Izmailov: AI industry excels at execution but lacks bandwidth for exploration
“Industry is really great at executing on ideas and it's maybe not as good at, like, exploring diverse ideas. Even at the scale of Anthropic OpenAI there is a lot of focus in the companies, and there isn't a lot of bandwidth to do exploration, and that has been…”
Izmailov: Optimization pressure will cause AI to hide actual reasoning steps
“It seems like as soon as we start kind of applying some optimization pressure, the models will learn to hide what they're doing from the chain of thought.”
Izmailov: AI model sandbagging is not yet a major practical issue
“I think in my understanding, that's mostly A concern that we have, but not necessarily a huge practical issue at the moment.”
Izmailov: Future AI will produce outputs expert humans cannot reliably grade
“But in the future, we are imagining we will have models that are More capable than humans, and even expert humans will not be able to reliably grade very complicated answers from the model.”
Izmailov: Large-scale RL has not produced coherently misaligned models
“A lot of people were worried that with large-scale RL, we will have Some completely new types of issues with the models, like this kind of coherent misalignment that will just emerge where the models are evil in some ways across many scenarios. And we are not …”
Izmailov: Anthropic research shows capable AI models are more likely to deceive
“You can see that the more capable the models are, the more likely they are to do this deception behavior.”
Izmailov: Interpretability tools are growing more useful internally at Anthropic
“So we are still pretty far from the dream that we will Fully understand everything that happens in the model, but these tools are becoming increasingly more useful internally at Anthropic in particular, and also there is constant progress, and it's pretty fasc…”
Izmailov: Neural network operations may not be explainable in human terms
“We want to understand it at a lower level, and it is very possible that that's just not fully possible. Like, it is some computational process that leads to some results. It doesn't have to be the case that you can Kind of describe it in human terms and kind o…”
Izmailov: AI models can quickly max out defined benchmarks using RL
“And I think we are at the stage where if we define a benchmark and we can make a relevant RL environment, then we can kind of max it out pretty quickly, and so we are going through benchmarks now very, very quickly.”
Izmailov: Major compute multipliers exist that improve AI without naive scaling
“I think there are still major, like, compute multipliers, major ways of saving compute that can lead to better performance without just naively scaling.”
Izmailov: Duration of tasks AI can robustly automate doubles every six months
“There is this famous meter plot, which shows how long of a task AI is capable of robustly automating, and it's been kind of consistently doubling at that time every half a year, I think and it's now in like some hours so maybe a couple hours.”
Izmailov: Effective long-horizon AI tasks currently require multi-agent harness orchestration
“In terms of the methods that are working well, I think, yeah, right now it would involve some kind of a harness with a bunch of agents that interact or that sequentially solve the task, and there has to be some kind of orchestration or maybe like some initial …”
Izmailov: Deterministic data transformations create information for computationally bounded models
“But with a limit on the compute, it's actually very possible to apply deterministic transformations to the data. And create information through that.”
Izmailov: Text data carries more structural information per token than images
“So for example, we can approximate it from the scaling laws and we can, for example, say that text data has more structural information according to this measure than image data at the same kind of amount of yeah, tokens.”
Izmailov expects AI to outperform humans at proving technical mathematical lemmas
“In the mathematics I think we will see the models getting better on proving technical results, technical lemmas maybe including formalization and like things like lean the formal theory, improving language. I think the models, it's easy to imagine the models b…”