May 9, 2024 · 1h 22m · big-technology
Anthropic's Co-Founder on AI Agents, General Intelligence, and Sentience — With Jack Clark
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Anthropic co-founder Jack Clark joins the Big Technology Podcast to explore the evolution toward autonomous AI agents, the compute and semiconductor bottlenecks shaping the industry, Anthropic's founding philosophy, and the safety evaluations required to govern frontier general intelligence.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Alex holds 27.5% of the talking time here. How this is scored →
Alex as informed peer 4.8 Guest teaching 5.5 Guest disagreement 1.3 Alex pushing back 2.6
speaking balance: gold is Alex, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:16:56 Challenging unconditional open-source release policies
Jack expresses willingness to have a direct, pugnacious debate with Meta or other advocates who oppose minimal security evaluations before open-sourcing powerful models.
Hardest push from Alex ▶ 36:35 Questioning arm's-length multi-billion-dollar cloud dealsAlex challenges the premise of Anthropic maintaining a distant partnership while taking billions from direct competitors Google and Amazon.
Biggest teaching moment ▶ 1:00:47 Reframing AI risk away from p-doom to systemic cyber catastropheJack educates the host by dismissing conventional p-doom framing and outlining the real-world mechanics of autonomous agent cyber vulnerabilities cascading across physical infrastructure.
Alex holds their own ▶ 40:34 Synthesizing market realities and chip architecture competitionAlex cites reporting from CNBC covering Google's Axion and Intel's Gaudi 3, demonstrating his own command of the semiconductor competitive landscape.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Alex as informed peer | Guest teaching | Guest disagreement | Alex pushing back | Why |
|---|---|---|---|---|---|---|
| Defining General Intelligence and Anthropic's Mission | 4 | 6 | 1 | 2 | Alex opens with broad industry framing around competitive dynamics and compute costs, while Jack reframes general intelligence away from human equivalence toward autonomous deep research across complex domains. | |
| The Evolution from Static LLMs to Autonomous AI Agents | 5 | 6 | 1 | 3 | Alex questions why existing context windows cannot already achieve general intelligence, prompting Jack to explain the critical distinction between passive task completion and agentic interaction. | |
| Multimodal Understanding, Context Windows, and Long-Term Memory | 4 | 6 | 1 | 3 | Alex probes the technical friction preventing persistent model memory, and Jack details the architectural tradeoffs between expensive short-term context windows, database tool use, and long-term storage. | |
| The Power and Mystery of Next-Token Prediction | 5 | 5 | 2 | 3 | Alex brings up Yann LeCun's physical reasoning critique, leading Jack to defend next-token prediction as an unexpectedly scalable paradigm that continues to confound traditional AI theoreticians. | |
| Emergent Capabilities, Mechanistic Interpretability, and Frontier Evals | 5 | 7 | 1 | 2 | Alex asks how to test for genuine emergent reasoning versus memorization, and Jack explains Anthropic's mechanistic interpretability and national security red-teaming evaluations on non-public data. | |
| The Convergence of Reinforcement Learning and Language Models | 6 | 5 | 1 | 2 | Alex demonstrates solid domain knowledge regarding self-supervised learning versus reinforcement learning, and Jack explains how RLHF and RLAIF merged the paradigms to produce modern conversational models. | |
| The Enterprise Business Case and Internal 'Claudification' | 5 | 5 | 1 | 3 | Alex challenges the return on investment of massive generative AI funding, prompting Jack to compare the current enterprise adoption phase to the early electrification of industrial factories. | |
| Consumer Utility, Mitigating Hallucinations, and Web Connectivity | 4 | 5 | 1 | 2 | Alex inquires about hallucination reduction and web browsing integration, while Jack outlines Anthropic's calibration research and security constraints regarding active web retrieval. | |
| Strategic Partnerships with Amazon, Google, and Frontier Safety | 5 | 5 | 2 | 4 | Alex pushes Jack on whether receiving billions from both Google and Amazon while maintaining an arm's-length competitive posture is viable, and Jack explains the strategic rationale of frontier safety competition. | |
| The Semiconductor Race, Nvidia's Dominance, and Compute Demand | 6 | 5 | 1 | 2 | Alex brings reporting context from CNBC regarding Nvidia's moat against custom ASICs, and Jack contextualizes the hardware shortage as a structural supply constraint driving massive capital requirements. | |
| AI Agents as the Next Operating System and Apple's Challenge | 5 | 4 | 1 | 2 | Alex and Jack explore how autonomous AI agents could disintermediate the mobile app store paradigm, creating a fundamental platform challenge for Apple's tightly controlled ecosystem. | |
| Mid-Roll Break: Teasing Anthropic's Origin Story | 4 | 6 | 1 | 2 | Alex asks Jack to verify or correct the narrative of Anthropic's spin-out from OpenAI, and Jack provides an insider historical account of scaling law discoveries and the development of GPT-2 and GPT-3. | |
| Deconstructing the Effective Altruism Connection and Internal Culture | 5 | 5 | 2 | 3 | Alex presses on the perceived connection between Anthropic and the Effective Altruism movement, and Jack clarifies the company's pragmatic, empirical culture versus ideological caricatures. | |
| AI Safety, Catastrophic Cyber Risks, and the Sentience Debate | 5 | 6 | 2 | 3 | Alex probes Jack's probability of doomsday existential risk, and Jack pivots from sci-fi p-doom tropes to concrete catastrophic threats like self-propagating autonomous ransomware and cyber infrastructure failure. | |
| Reflecting on Introspection, User Privacy, and Inner Model Mechanics | 4 | 5 | 1 | 3 | Alex asks whether Jack believes Claude displays sentience, leading Jack to share a personal experience where Claude's therapeutic introspection prompted an extended reflective walk. | |
| Assessing the Economic Impact of AI on the Labor Market | 5 | 6 | 1 | 2 | Alex inquires about labor displacement and brings up Jon Stewart's critiques of regulation, while Jack outlines the ongoing international third-party evaluation regime forming in the US, UK, and EU. | |
| The Open-Source AI Debate and Responsible Deployment Standards | 5 | 6 | 3 | 3 | Alex asks about Anthropic's stance on open source, prompting Jack to argue that while most models should be open, frontier models crossing clear national security thresholds must pass rigorous pre-deployment evaluations. | |
| Research on AI Persuasion and Democratic Election Guardrails | 4 | 6 | 1 | 2 | Alex asks about Anthropic's newly published research on model persuasion, and Jack reveals empirical findings showing that scaled LLMs match human-level capability in shifting human beliefs. |
Statements from this episode (43)
Clark: Frontier AI training costs jumped from thousands to hundreds of millions
“It costs, you know, tens of millions, maybe hundreds of millions of dollars to train these things now. Back in 2019, it cost tens of thousands of dollars.”
Clark: Compute represents the vast majority of frontier AI training costs
“It's mostly compute. Talent, talent matters and data matters. The vast majority of the expense here is on compute to train the models.”
Clark: General intelligence requires real-world interplay that current AI lacks
“General intelligence comes from, like, interplay with the world around you and interaction with it, and today's systems don't really do that at all. To the extent they do, it's kind of a fiction, and we need to teach them how to do that.”
Clark: AI systems capable of sequential actions will emerge in 2024
“I think this year you're not going to see the exact thing I described, but you're going to see systems that start to take multiple actions. You know, you may have heard lots of guests talk about things like agents. I think what an agent is, is a language model…”
Clark: Powerful AI agent systems will arrive in three to five years
“I would be pretty surprised if in the order of like three to five years, we didn't have quite powerful things that seem somewhat similar to what I've described. But I also guarantee you, we will have discovered some ways in which these things seem wildly dumb …”
Clark: Anthropic is developing audio and video capabilities for Claude
“Claude can now see images. Obviously we're working on other so-called modalities as well. You know, it would be nice for Claude to be able to listen to things. Be nice for Claude to understand movies. All of that stuff is going to come in time.”
Clark: AI models need long-term storage instead of massive context windows
“Well, our AI systems today are kind of operating with short term memories that are millions of numbers in length, and it feels very unintuitive. Ultimately, we want them to instead be able to bake stuff into some kind of long term storage, and that's going to …”
Clark: Anthropic's internal engineering philosophy is 'do the dumb thing that works'
“So at Anthropic, we have this public value statement, which is do the simple thing that works. But actually internally, we sometimes say an even cruder version, which is do the dumb thing that works”
Clark: Advancing AI will probably require few complex breakthroughs beyond simple scalable ideas
“And I guess my naive view is the amount of things we'll need to do that are extra special will probably be quite small. And the challenge is coming up with simple ideas like next token prediction, but scale for probably other simple ideas we need to figure out…”
Clark: Anthropic operates Frontier Red Team for national security evals
“Anthropic has a line of work on what we call the Frontier Red Team, where we are doing national security relevant evaluations.”
Clark: Passing government evals proves Claude has emergent reasoning capabilities
“If Claude can figure out things and trigger, like, threshold points on those evals, we know something creative is happening. Because Claude has reasoned its way to things that the government has believed are very hard to reason your way to unless you have acce…”
Clark: Scaling RL compute will unlock major new AI capabilities
“Everyone is trying to figure out how they can spend more and more of their compute on reinforcement learning, because I think everyone has this intuition that the more RL you add, the more sophisticated you're going to be able to make these things, and a lot o…”
Clark: Consumer AI subscriptions will become a massive market like Netflix
“I think there's going to be this very large growing business of basically a subscription model where people will have a personal AI or multiple AIs that they use, just like you or I might have a Netflix account or whatever.”
Clark: AI's true value lies in companies built from scratch around AI
“I think that where the value is going to come from will be from that second class of businesses, which were just in the early innings of sort of helping to build together.”
Clark: Anthropic runs internal 'Claudification' project across its workflows
“We have a project internally called Claudification. Everything at Anthropic has Claude in it at some point. And one of the ideas of Claudeification is just get us all to use this stuff well. I talked about my colleague, Catherine, but there are many examples w…”
Clark: Anthropic will always keep an accessible consumer product
“We're always going to have some Like, top of funnel, easy to access consumer thing, because we just can't ignore how useful this is to people, you know, and useful it is to writers especially.”
Clark: Live web search connectivity is 'definitely coming' to Claude
“And on the web question, we're working on it. There's a bunch of Kind of computer security stuff to work through and some safety things, but that's definitely coming.”
Clark: Anthropic models sometimes exhibit situational awareness during testing
“We've done some self-awareness tests. There have been a few, but we've definitely done this, and yeah, sometimes they have what you call situational awareness. One of the things my colleagues in Interpretability are working on is a really good test for that, b…”
Clark predicts winning in AI will force rivals to adopt Anthropic safety practices
“And I think the best thing that we can do for the ecosystem is compete really, really hard with kind of everyone in it and win. And that's going to cause people to adopt a load of our safety staff to try and compete against us.”
Clark: Anthropic builds on Amazon Trainium, Google TPU, and Nvidia chips
“I can't get too much into the specifics, but I can just say we've sort of publicly stated that we're Working on both Tranium chips and also TPU chips. We also work on NVIDIA chips as well.”
Clark: Anthropic is grabbing all available chips to meet Claude 3 demand
“I mean, we ourselves
have been experiencing this where we've been, you know, very successful with Claude Free, and we've been you know, going and doing the supermarket sweep to grab as many chips as we can to, like, serve all the customers we have. The chip …”
Clark: Developers will build and release open-source AI agents
“People are definitely going to build, like, open source agents and release them as well, and we're going to have to contend with that where the environment of the internet will be changed by this in a bunch of hard to predict ways.”
Clark: AI is especially challenging for Apple due to its unstable nature
“Well, I was going to say that I think one thing that's challenging about AI is that we're in this giant experimental phase. And I think when you think of like, Experimental and like people don't have a clear notion of what to do. You don't think of us like pre…”
Clark: OpenAI deliberately downplayed GPT-3's release to test public discovery
“We actually tried to lowball the system in that we published a research paper called like language models are few shot learners. I don't think we even tweeted about it. We tried to like public publish it publicly, but also be like very quiet and see, see how q…”
Clark: Anthropic left OpenAI to pursue coherent safety bets, not from hostility
“And it's not to say that there's any particular. Like distaste for safety there. It's more that you had. We had, like, a very specific view, and other people had views, so you were gonna win some, lose some, and then we realized, well, we could just do this to…”
Clark: Dario Amodei's early Constitutional AI concept sounded completely crazy
“On, I think, like, week four we were talking about RL and language models, and Jared was like, oh, Dario says we're just gonna write a constitution for the AI and it'll just follow that. And I remember being like, that's completely crazy. Why would this ever w…”
Clark: AI Safety Talent Pool Heavily Overlaps With Effective Altruism
“Of the group of people in the world that have spent a long time thinking about AI, are really good at math and science, and have worried about some of the safety issues, there is a huge overlap with this community of people called effective altruists, and so s…”
Clark: Anthropic is not driven by Effective Altruism ideology
“The organization is much more, Like oriented around trying to build some useful AI staff, prove that it works in the world and be very sort of pragmatic. We're not driven by some kind of like EA ideology. And in the early days we hired quite a few people from …”
Clark: Scaled AI integrated into critical infrastructure risks catastrophic cascading failures
“I think that if you really scale up AI systems, and you plug them into important parts of the world, and they go wrong, the effects could be extraordinarily, like, bad and catastrophic, in the sense of some,
Cascading emergent problem.”
Clark: Unregulated AI market cannot guarantee developers won't cut corners
“I think if we built, if we do nothing on policy or regulation, we're sort of gambling that everyone is going to be reasonably responsible and not cut corners. And I think in a really like fast moving, crazy technology market, like AI, you aren't really guarant…”
Clark: Claude 3 is far more complex and weird to explore than previous LLMs
“The only claim I'm going to make is it certainly got a lot more complicated and weird to explore than previous systems or other language models that have been developed.”
Clark: AI sentience could become a field of study with systemic risks
“I'm saying that we may enter the weird zone where that becomes a thing that people study. And I think that if like sentience is a thing, you could imagine like weird versions of it leading to certain types of misuses or problems in the system as well.”
Clark: Anthropic does not train models on Claude user conversations
“No, no, no, no. That is not a thing that we do at all. Yeah.”
Clark: AI-native startups will do much more with fewer people
“My bet is that you're going to see new companies get formed, which do a lot more with a lot less in terms of people. They're going to figure out how to be, like, much smarter and perform a lot better than that than equivalently scaled companies that don't use …”
Clark: AI will not cause drastic job automation in next few years
“It'll certainly change jobs in a bunch of ways. But it's not going to be some instant, like, or drastic automation thing, at least in the next few years. It's going to be more like augmenting jobs or making people a lot more effective.”
Clark: US and UK AI Safety Institutes Are Testing Frontier AI Systems
“The US and UK are not, don't have regulatory powers. Will they be third parties that test out systems like Claude or ChatGPT or Gemini for national security risks and hold companies accountable to them? Yes. Like I'm in discussion with them today.”
Clark: Governments Will Punish AI Labs That Ignore Severe Safety Findings
“And you can bet, you know, I haven't spoken to others, but I can bet that if they find severe risks and we don't do anything about it and we deploy our systems, they will come for us in, in a pretty, pretty, pretty clear way.”
Clark: All AI models should undergo pre-deployment third-party safety testing
“We need some set of tests administered by a third party for things that people would view as legitimate, like national security risks or what have you, and systems, whether proprietary like ours or otherwise, Should go through those tests before they're deploy…”
Clark: Open-sourcing Anthropic's Claude 3 today would be 'broadly fine'
“I think you could release, like, pretty much everything as open source today. I think maybe even clawed free, and things would be fine. Like, it would be a little spicy, maybe surprising stuff would happen, but probably broadly fine.”
Clark: AI models triggering national security tests should not be open-sourced
“I do expect that if we end up in a world where, like, we trigger a national security test it would be very hard for me to make the claim that that system which has triggered that test should be released as open source. Like, these things, like, I can't reconci…”
Clark: Anthropic previously put too much safety training into its models
“Actually, like, we've gone through this at Anthropic. We've, like, over put too many of the safety ingredients in some of our models before, and it's led to them seeming annoying to people.”