Cloudflare blocks AI bots by default on monetized or paywalled web pages
Matthew Prince · Can We Save The Web From AI? — With Cloudflare CEO Matthew Prince · Aug 13, 2025 · at 33:31
Cloudflare CEO Matthew Prince explains the technical heuristic Cloudflare uses to restrict AI scrapers on publisher sites.
0:00 / 0:25exact quote · 25.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“What we've done is we've actually tried to identify with real narrow precision. What are those pages that are on the web that have some indication that they are going to be monetized. And generally that is look at, is it behind a paywall or is it, does it have some sort of an ad unit on it? Like a banner ad or some sort of ad that's there. If we detect that, then we're blocking it by default”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Matthew Prince
Prince: Earning traffic from OpenAI is 750x harder than Google historically
“In OpenAI's case, it's 750 times harder to get traffic than it was from Google just 10, 10, 10 years ago. In the case of something like Anthropic, it's 30,000 times more, more difficult to get that traffic.”
Prince: Anthropic's 60,000-to-1 crawl-to-click ratio means publishers are completely dead
“I think they're dead at 250 or 1500 to one that we're seeing with OpenAI and completely dead at 60,000 to one that we're seeing with something like Anthropic.”
Prince: Cloudflare will block Google scrapers for publishers starting in September
“And what we have said at cloudflare is that by mid September, we're gonna treat Google the same way on. If you have an, you know, ads, if you're a business that's ad supported, subscription supported, then we're going to restrict Google from being able to acce…”
Cloudflare announces hard-blocking AI crawlers unless they pay creators
“And then just today on July first, we announced that we are hard blocking the AI crawlers unless they will actually compensate Content creators for the content that they're creating.”