Cloudflare CEO Matthew Prince explains how Google bundles multiple crawling functions together, preventing publishers from selectively blocking AI ingestion.
“Google's crawler does five different things, at least. One is it checks if you have an ad on a page. It makes sure that if you're putting an ad for Procter & Gamble, Procter & Gamble product up, it's not against a pornographic site or something like that. So it does brand safety checks. The second thing that it does is crawls to index for traditional search, the 10 blue links that are out there. The third is that it crawls to create answers that are in the answer box. The fourth is that it crawls to create answers that are in the AI overview, the newer thing that they've rolled out. And the fifth is that it crawls in order to ingest content In order to put it into Gemini.”
quote is from the automated transcript, cleaned for reading:
filler sounds and stutters are removed, nothing is rephrased. names can be misheard
(the analysis reads context, assessments check outside sources). how →
More from Matthew Prince
AssertionNot checkable as stated
Prince: Earning traffic from OpenAI is 750x harder than Google historically
“In OpenAI's case, it's 750 times harder to get traffic than it was from Google just 10, 10, 10 years ago. In the case of something like Anthropic, it's 30,000 times more, more difficult to get that traffic.”
Matthew PrinceAug 13, 2025▶ 4:02Can We Save The Web From AI? — With Cloudflare CEO Matthew Prince
AssertionSupported
Prince: Anthropic's 60,000-to-1 crawl-to-click ratio means publishers are completely dead
“I think they're dead at 250 or 1500 to one that we're seeing with OpenAI and completely dead at 60,000 to one that we're seeing with something like Anthropic.”
Matthew PrinceAug 13, 2025▶ 8:28Can We Save The Web From AI? — With Cloudflare CEO Matthew Prince
Disclosure
Prince: Cloudflare will block Google scrapers for publishers starting in September
“And what we have said at cloudflare is that by mid September, we're gonna treat Google the same way on. If you have an, you know, ads, if you're a business that's ad supported, subscription supported, then we're going to restrict Google from being able to acce…”
Matthew PrinceSep 2, 2026▶ 30:05Cloudflare CEO: We're Ready To Block Millions of Websites From AI — With Matthew Prince
Disclosure
Cloudflare announces hard-blocking AI crawlers unless they pay creators
“And then just today on July first, we announced that we are hard blocking the AI crawlers unless they will actually compensate Content creators for the content that they're creating.”
Matthew PrinceAug 13, 2025▶ 4:42Can We Save The Web From AI? — With Cloudflare CEO Matthew Prince
AssertionSupported
Prince: Automated web traffic surpassed human traffic for first time in 2026
“In May of 2026, you won't believe it, but we now see that automated traffic is past human traffic for the first time in the history of the internet.”
Matthew PrinceSep 2, 2026▶ 2:16Cloudflare CEO: We're Ready To Block Millions of Websites From AI — With Matthew Prince
Opinion
Prince: App stores and Facebook damaged the open web
“And the app store in some ways was an enormous threat to the web. Like it took people away from the web. It was probably one of the biggest, between it and Facebook were like the two biggest forces that really made the web less interesting.”
Matthew PrinceSep 2, 2026▶ 6:44Cloudflare CEO: We're Ready To Block Millions of Websites From AI — With Matthew Prince
Made with StarZero
Turn any episode into a week of clips.
This entire site, over 300 episodes transcribed, diarized, checked and made playable,
runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the
moments worth sharing, cuts them, captions them, and reframes them for every feed.
We use essential cookies to make the site work. With your permission we
also use analytics cookies (Google Analytics and Mixpanel) to understand
usage and improve StarZero. See our Cookie Policy.
