Aug 13, 2025 · 51m · big-technology
Can We Save The Web From AI? — With Cloudflare CEO Matthew Prince
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Cloudflare CEO Matthew Prince and host Alex Kantrowitz discuss how generative AI disrupts the open web's traditional search-and-referral economy, highlighting Cloudflare's technical initiatives—such as HTTP 402 paywalls and publisher coalitions—to establish fair compensation for human creators.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Alex holds 22.4% of the talking time here. How this is scored →
Alex as informed peer 5.6 Guest teaching 5.7 Guest disagreement 2.0 Alex pushing back 3.9
speaking balance: gold is Alex, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 19:56 Prince dismisses the subscription defense as a temporary illusion
Prince firmly rejects Kantrowitz's argument that creators can simply retreat to direct newsletters and podcasts, asserting AI agents will ingest and summarize that content just as easily.
Hardest push from Alex ▶ 19:13 Kantrowitz challenges the premise that traffic is still viable for mediaKantrowitz directly challenges Prince's premise by citing a working journalist who abandoned web traffic dependency long ago in favor of direct reader subscriptions and events.
Biggest teaching moment ▶ 35:08 Prince deconstructs the five hidden jobs of Google's crawlerPrince educates Kantrowitz on technical crawler nuances, demonstrating how Google combines brand safety checks, web indexing, answer boxes, AI overviews, and Gemini training into one crawler.
Alex holds their own ▶ 24:09 Kantrowitz frames AI content proliferation as a web DDoSKantrowitz demonstrates technical expertise by constructing an argument that human-plus-bot content generation acts like a DDoS attack that destroys the scarcity Cloudflare's marketplace relies on.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Alex as informed peer | Guest teaching | Guest disagreement | Alex pushing back | Why |
|---|---|---|---|---|---|---|
| Comparing Search Crawl Ratios Across Google, OpenAI, and Anthropic | 4 | 7 | 1 | 1 | Kantrowitz opens by asking Prince to detail historical search crawl ratios and traffic returns. Prince responds with comprehensive proprietary Cloudflare telemetry, contrasting Google's 18-to-1 crawl ratio with OpenAI's 1,500-to-1 and Anthropic's 60,000-to-1. | |
| The Urgency Behind Media Panic During AI's Infancy | 6 | 5 | 2 | 5 | Kantrowitz pushes back on the panic narrative, pointing out that Anthropic has low adoption and OpenAI's 500 million weekly users pale in comparison to total web traffic. Prince justifies the urgency by citing Google search volume decline data revealed during the Apple antitrust trial. | |
| Balancing User Experience With Content Creation Incentives | 6 | 5 | 2 | 6 | Kantrowitz challenges the publisher grievance by arguing direct AI answers offer superior consumer user experience over clicking low-value SEO sites. Prince concedes the UX advantage but stresses that cannibalizing original content creators destroys the training fuel necessary for AI models. | |
| Distinguishing Model Training From Real-Time Data Retrieval | 6 | 6 | 2 | 4 | Kantrowitz distinguishes between model training crawls and real-time retrieval queries, arguing training ingestion poses less direct competition to search. Prince clarifies that both activities impose infrastructure costs and extract uncompensated IP value from creators. | |
| Legal Fair Use Debates and Technical Control Mechanisms | 7 | 6 | 3 | 3 | Kantrowitz cites fair use legal precedents and Wikipedia's escalating server costs due to AI scrapers. Prince argues legal battles are too slow and inconclusive, insisting technical access controls and pricing mechanisms must precede legal resolutions. | |
| The Risk of Oligopoly Knowledge Institutions and Open Web Vision | 6 | 5 | 2 | 4 | Kantrowitz shares conversations with data labeling specialists, questioning whether the open web might be superseded by direct publisher-to-LLM data pipelines. Prince warns this would recreate a Medici-style oligopoly where five tech giants control captive academic and journalistic institutions. | |
| Moving Past Subscriptions to Fill Knowledge Gaps | 6 | 7 | 4 | 6 | Kantrowitz challenges Prince by quoting a journalist peer who argues sustainable media relies on direct subscriptions and podcasts rather than search traffic. Prince counters that AI audio synthesizers and summary agents will bypass subscriptions just as easily, citing Spotify founder Daniel Ek's model of monetizing search demand gaps. | |
| Synthetic Content Proliferation and the Premium on Human Insight | 7 | 6 | 3 | 6 | Kantrowitz raises the threat of synthetic content saturation acting as a distributed denial-of-service on human content value. Prince uses the film Multiplicity to illustrate model collapse from synthetic training loops and highlights niche human expertise like hyper-local weather forecasting. | |
| Cloudflare's Pivot From State Hackers to AI Scraper Mitigation | 5 | 5 | 2 | 4 | Prince recounts Cloudflare's pivot from defending against state-sponsored hackers to mitigating aggressive AI scraper traffic. Kantrowitz questions whether an AI-mediated internet renders Cloudflare's traditional website protection services redundant, prompting Prince to note that Cloudflare protects 80 percent of AI companies. | |
| Modernizing Web Protocols With HTTP 402 Payment Required | 5 | 6 | 1 | 2 | Prince explains the shortcomings of robots.txt and outlines Cloudflare's revival of the HTTP 402 'Payment Required' protocol to enable paywalled scraping. Kantrowitz tracks the technical mechanics and implications for dynamic market clearing prices. | |
| Publisher Coalition Launch and Bilateral AI Licensing Deals | 6 | 6 | 2 | 4 | Kantrowitz notes the publisher coalition launch at One World Trade Center and asks whether bilateral AI licensing deals are merely payoffs to avoid litigation. Prince compares current AI licensing dynamics to the transition from Napster to iTunes and Spotify, explaining why free-rider access undermines bilateral agreements. | |
| SEO Implications, Regulatory Pressures, and the Plateauing Web | 5 | 5 | 1 | 4 | Kantrowitz asks whether blocking AI scrapers risks damaging publishers' traditional search engine optimization traffic. Prince explains that standard search indexing remains unblocked and expects regulatory mandates to force search engines to decouple index crawling from AI training. | |
| AI Cybersecurity Dynamics and Long-Term Productivity Outlook | 4 | 5 | 1 | 1 | Kantrowitz closes with inquiries into AI use by cybercriminals and Prince's timeline for artificial general intelligence. Prince argues defensive data advantages keep security ahead of attackers while asserting that ninety-nine cents of every AI venture dollar is currently wasted. |
Statements from this episode (32)
Prince: Google search query volumes are declining for the first time
“What we're seeing though, is that for the first time in history, searches across the major search engines, Google in particular, are actually on the decline.”
Prince: Google crawl rate has stayed consistent over 10 years
“And we have very reliable data at Cloudflare going back 10 years, looking just at Google and the metric that has stayed very consistent over time is how much Google crawls the web. They've actually crawled at a very consistent rate over the last 10 years.”
Prince: Getting a referral click from Google is 10x harder today
“Today, it's almost 10 times as hard to get a click, to get a visitor from Google to your site.”
Prince: 75% of Google search queries are answered without a click
“In fact, about 75% of queries to Google now get answered on Google itself.”
Prince: Earning traffic from OpenAI is 750x harder than Google historically
“In OpenAI's case, it's 750 times harder to get traffic than it was from Google just 10, 10, 10 years ago. In the case of something like Anthropic, it's 30,000 times more, more difficult to get that traffic.”
Cloudflare announces hard-blocking AI crawlers unless they pay creators
“And then just today on July first, we announced that we are hard blocking the AI crawlers unless they will actually compensate Content creators for the content that they're creating.”
Prince: AI-driven web will cause media consolidation and reduced content
“Again, I think that that's just going to mean that over time we'll have more consolidation in the media industry. We'll have less and less content. We'll have actually more salacious headlines as people are chasing The content that is, is left that's out there…”
Prince: Google's AI Overviews pushed its crawl-to-click ratio to 18:1
“The ratio 10 years ago of crawls from Google to clicks was two crawls, one click. Six months ago, it was up to six crawls one click, and that's all because of the answer box. What the AI overviews, which they've rolled out over that time, have done is they've …”
Prince: Anthropic's 60,000-to-1 crawl-to-click ratio means publishers are completely dead
“I think they're dead at 250 or 1500 to one that we're seeing with OpenAI and completely dead at 60,000 to one that we're seeing with something like Anthropic.”
Prince: Strangling publisher incentives will eventually degrade AI systems
“If we strangle off the business model of those places, if we strangle off the incentives for content to create for content creators to create content, then we're actually going to end up strangling the AI systems as well, because there's, if there's no content…”
Prince: AI crawling for training and RAG imposes bandwidth costs on publishers
“I think in both cases though, you're actually costing the content creator something. There is literally, they're paying for that traffic. They're paying for that load that the crawlers are Pulling off of it.”
Kantrowitz: AI bots caused Wikipedia's server costs to increase exponentially
“The server costs that they have to pay have increased exponentially, but those aren't human visitors. They're AI bots crawling Wikipedia.”
Prince: California courts issued contradictory rulings on AI training fair use
“We literally had two different California cases that came out on both sides of that issue. Is, Training on content fair use or not.”
Prince: Strangling creator incentives will ultimately hurt AI companies
“In order for AI to get better, you have to have original content. People have to be going out and creating that. And right now we're strangling off all of the incentives for that content creation, which not only hurts content creators, it will ultimately hurt …”
Prince: AI bots should pay heavily for content so humans read free
“Humans should get content for free again, because we've kind of paywalled way too much, frankly. And robots should pay a ton for it. Because again, every time a robot ingests something, it's in service of, Hundreds of thousands, if not millions of different hu…”
Prince: AI will eliminate creator subscription revenue by summarizing podcasts
“That will go away as well, because what will happen is the AI company will ingest the podcast and then summarize it on, on their page.”
Prince: AI will generate hyper-customized podcasts in personalized voices
“We're going to want to have hyper-customized podcasts that are in exactly the voice that we find the most reassuring and AIs are going to create that for us.”
Prince: Creators make millions fulfilling unserved audio search queries on Spotify
“There are content creators out there that are making tens of millions of dollars a year just creating content for those searches that don't have good results right now, because Spotify surfaces that list of things where they don't give those results.”
Prince: AI web scrapers brought down entire publisher websites with traffic
“Starting really 18 months ago is that those companies said, hey, I know we hired you in order to stop the Chinese hackers, but we have this new threat that's there... And it was only when we actually saw the data and we saw how AI companies were taking content…”
Cloudflare blocks AI bots by default on monetized or paywalled web pages
“What we've done is we've actually tried to identify with real narrow precision. What are those pages that are on the web that have some indication that they are going to be monetized. And generally that is look at, is it behind a paywall or is it, does it have…”
Prince: 80% of AI companies are Cloudflare customers
“80% of the AI companies are our customers of ours, and so, so we project them as well.”
Prince: Major legitimate companies completely ignore robots.txt
“And there's some just, there's some, even some big legitimate companies that completely ignore robots.txt.”
Prince: Google uses a single crawler for search, ads, and Gemini
“Google's crawler does five different things, at least. One is it checks if you have an ad on a page. It makes sure that if you're putting an ad for Procter & Gamble, Procter & Gamble product up, it's not against a pornographic site or something like that. So i…”
Prince: Cloudflare working with IETF and regulators on robots.txt extensions
“And so what we've proposed, and we're working with the IETF as well as regulators is extensions to robots.txt to give it that granularity.”
Prince: Cloudflare is adopting the HTTP 402 code to charge AI bots
“There's actually an original one of the protocols set out a four or two response and that response says payment required. And so we're actually tapping into that exact original specification to say when a robot tries to access, access a page, Where there's an …”
Prince: Crawler paywalls will create a dynamic marketplace for content
“And so that's the sort of thing where I think a marketplace over time can develop where original valuable content will get compensated and there will be a clearing price in the market once we have that scarcity that's created by that wall.”
Prince: AI content licensing fails if competitors get content for free
“What I know won't work is if open AI is paying for your content, but you're giving it away for free to everyone else.”
Prince: Global regulators will force Google to split its web crawler
“I am also a hundred percent certain that regulators are paying a ton of attention to this. And that around the world, you will see them force Google to split their crawler out into announcing exactly what it is doing.”
Prince: Open web content growth has plateaued amid fewer domain registrations
“By all the measures that we can see, it's actually kind of plateaued and is actually flattened out in terms of in terms of content. You see fewer domains getting registered. You see fewer new websites going online.”
Prince: Cybersecurity defenders will outpace attackers via superior AI data access
“At the end of the day, who wins in the AI risk is whoever has the access to the most data. And I just think that the good guys are always going to have access to a lot more data than the bad guys.”
Prince: 99 percent of AI spending is currently being lit on fire
“I believe today that 99 out of, cents out of every dollar spent on AI is just being lit on fire. But that one cent that's out there Is gonna generate real returns.”
Prince: AI support agents and lawyers lag coding due to lack of debuggers
“On the other hand, you know, I am, I, I'm still quite skeptical that the AI customer support agent that, that is a much harder problem or the AI lawyer that is a much harder problem because it's just harder to tell whether something actually worked or didn't. …”