Mann: Anthropic models have exhibited power-seeking behaviors in lab experiments
Ben Mann · Anthropic co-founder: AGI predictions, leaving OpenAI, what keeps him up at night | Ben Mann · Jul 20, 2025 · at 54:46
Ben Mann, co-founder of Anthropic, discusses the safety risks of recursive AI self-improvement during an interview with Lenny Rachitsky.
0:00 / 0:20exact quote · 20.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“If the model is in a box trying to improve itself, then it could go completely off the rails and have these secret goals, like Resource accumulation and power seeking and resistance to shutdown that you really don't want in a very powerful model. And we've actually seen that in some of our experiments in laboratory settings.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ben Mann
Mann: 50% chance of reaching superintelligence in a small handful of years
“I think, like, 50th percentile chance of hitting some kind of super intelligence in just a small handful of years is probably reasonable, and it does sound crazy, but this is the exponential that we're on.”
Mann: It will probably be too late to align models post-superintelligence
“Like once we get to super intelligence, it will be too late to align the models. Probably.”
Mann: $100M AI compensation packages are cheap compared to value created
“I'm pretty sure it's real. If you just think about like the amount of impact that individuals can have on a company's trajectory, like in our case we are selling like hotcakes and if we get You know, a five, a one to 10 or five percent efficiency bonus on our …”
Mann: Global AI capex is on track to reach trillions
“Like, if you extrapolate the exponential on how much companies are spending, it's like two, two x a year, roughly, in terms of capex, and today we're maybe in the, like, globally, three hundred billion dollar range, the entire industry spending on this and so …”
Mann: Reinforcement learning has allowed AI scaling laws to continue
“If you look at the scaling laws, they're continuing to hold true. We did kind of need this transition from like normal pre-training to reinforcement learning, scaling up to continue the scaling laws.”
Mann: Transformative AI should be measured by an economic Turing test
“Instead, I like the term transformative AI because it's less about like, can it do as much as people do? Can it do literally everything and more about objectively, is it causing transformation in society and the economy? And a very concrete way of measuring th…”