Sep 2, 2026 · 51m · big-technology
Cloudflare CEO: We're Ready To Block Millions of Websites From AI — With Matthew Prince
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Big Technology Podcast, host Alex Kantrowitz interviews Cloudflare CEO Matthew Prince about the structural shift of automated AI traffic surpassing human browsing and the urgent steps required to protect the open web. Prince explains Cloudflare's decision to block Google AI scrapers by default, examines infrastructure scaling bottlenecks, and analyzes the severe economic threats artificial intelligence poses to independent publishers and small businesses.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Alex holds 22.4% of the talking time here. How this is scored →
Alex as informed peer 5.1 Guest teaching 5.6 Guest disagreement 2.8 Alex pushing back 3.4
speaking balance: gold is Alex, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 29:30 Prince targets Google's monopoly and crawler dominance
Prince forcefully rejects Google's privileged search crawler status, insisting Google is just another AI company that must pay for content and declaring Cloudflare's intent to block them by default.
Hardest push from Alex ▶ 45:53 Kantrowitz challenges Prince's small business pessimismKantrowitz directly counters Prince's thesis by arguing that existing brand advertising already favors incumbent giants with deep pockets, whereas AI agents could discover superior small business products on merit.
Biggest teaching moment ▶ 16:50 Prince calculates the computational impossibility of agent scalingPrince educates the host on underlying hardware realities, demonstrating that running one basic container per knowledge worker would require over 40 times annual global CPU production.
Alex holds their own ▶ 22:45 Kantrowitz quotes precise crawl-to-click multiplesKantrowitz shows deep preparation by detailing exact historic crawl-to-click ratios from Google, OpenAI, and Anthropic to establish how dramatically the publisher value proposition has eroded.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Alex as informed peer | Guest teaching | Guest disagreement | Alex pushing back | Why |
|---|---|---|---|---|---|---|
| Surpassing Human Traffic and Refuting Dead Internet Theory | 4 | 6 | 3 | 2 | Kantrowitz introduces Cloudflare's finding that bot traffic surpassed human traffic and brings up Dead Internet Theory. Prince rejects the dead internet framing with proprietary Cloudflare data, explaining that AI-assisted human creation is actually driving a web resurgence similar to the mid-2000s. | |
| Creator Democratization Versus App Store Hegemony | 6 | 5 | 4 | 6 | Kantrowitz pushes back on Prince's optimistic view of web creation by citing App Store data showing falling reviews and stagnant active users. Prince defends his premise by arguing that closed app stores threatened the open web and that AI developer tools are re-empowering individual creators. | |
| Agent Crawling Volume and Physical Infrastructure Constraints | 5 | 7 | 2 | 3 | Kantrowitz asks for a breakdown between training, agents, and test-time retrieval queries. Prince educates him on the impending 1000x bot traffic surge, using the near-collapse of European internet infrastructure during COVID-19 and WordPress hosting economics to demonstrate physical constraints. | |
| Architectural Scaling and Distinguishing Web Content Types | 5 | 7 | 2 | 4 | Kantrowitz probes what justifies the 1000x estimate. Prince details the computational impossibility of running knowledge-worker agents on standard containers without exceeding 40 times annual global CPU production, explaining the architectural shift toward lightweight workers and isolates. | |
| The Death of Referral Traffic and High-Speed Micropayments | 7 | 6 | 3 | 3 | Kantrowitz demonstrates strong mastery by recalling detailed crawl-to-click ratios across Google, OpenAI, and Anthropic from their prior interview. Prince explains that better AI quality has halved referral click-throughs and details the need for high-throughput micropayments infrastructure. | |
| Cloudflare's Default Block on Google Scrapers for Publishers | 7 | 6 | 5 | 4 | Kantrowitz cites reporting from The Wall Street Journal and search query metrics to frame the standoff between publishers and Google. Prince adopts a combative stance toward Google's crawling monopoly, breaking the news that Cloudflare will restrict Google crawlers by default for ad-supported publishers in mid-September. | |
| Mid-Roll Teaser: Examining the Threat to Small Businesses | 4 | 7 | 2 | 2 | Kantrowitz sets up Prince's thesis regarding the existential risks of agentic web browsing for small businesses. Prince delivers an in-depth explanation of how unemotional, deeply researched AI agents will bypass local and emotional branding in favor of established data-rich mega-corporations. | |
| Cryptographic Attestations and the AI Economic Divide | 7 | 6 | 4 | 7 | Kantrowitz offers a sharp counterargument, contending that traditional branding already favors big budgets and that agentic research could level the playing field for better niche products. Prince responds by highlighting the cold-start problem of verifiable reputation and the broader divide between rich and poor internet users. | |
| Concluding Thoughts and Future Outlook on Internet Architecture | 1 | 0 | 0 | 0 | The episode wraps up with friendly closing pleasantries and a commitment to revisit the conversation in a year. |
Statements from this episode (22)
Prince: Automated web traffic surpassed human traffic for first time in 2026
“In May of 2026, you won't believe it, but we now see that automated traffic is past human traffic for the first time in the history of the internet.”
Prince: Automated Web Traffic Will Be 1,000x Human Traffic Within Five Years
“And if we just assume the current trends, what's, what's blows my mind is that if you just fast forward five years from now we estimate now that automated traffic online will be a thousand times what human traffic is.”
Prince: Future Web Consumption Will Be Heavily Mediated by AI Interfaces
“And instead of it making the internet worse, I actually think the internet's getting better right now. It's just the interface that we're all going to consume it through is going to be very different. It's going to be much more mediated through some sort of an…”
Prince: The tech ecosystem will move away from the app store model
“I do think that we're going to move more and more away from kind of an app store model. Because that's not going to be the interface where we reconnect with it.”
Prince: App stores and Facebook damaged the open web
“And the app store in some ways was an enormous threat to the web. Like it took people away from the web. It was probably one of the biggest, between it and Facebook were like the two biggest forces that really made the web less interesting.”
Prince: Compensating creators via AI access will revitalize the web
“I think now putting things on back on the web and then making sure that they're accessible across the different AI systems in a way that makes sense for creators and make sure that creators get compensated is going to be something which actually I think will r…”
Prince: Defining internet monetization is tech's most important question
“Over the next five years, figuring out what the business model of the next 30 years of the internet, like, that's the most interesting and important question in technology today, by far.”
Prince: Autonomous agent-to-agent traffic will grow into a major web category
“I think at the same time, you've got sort of agents that are operating on their own. That's probably the smallest of the categories right now. I think that that will grow and become a pretty large category, but right now that's sort of Agent to agent kind of c…”
Prince: European internet neared collapse as traffic doubled during COVID
“The internet, European internet came this close to completely collapsing in COVID because in COVID over a period of two weeks, Internet traffic doubled, and it especially doubled for video streaming, which was, ah, which was a very high bandwidth.”
Prince: WordPress's underlying PHP technology is too inefficient for AI scale
“WordPress, most popular platform for hosting content. That's out there is running a programming language called PHP, not the most efficient programming language.”
Prince: Universal containerized AI agents require 40x global annual CPU production
“If every one of those agents ran in a traditional container, like, you know, which is not even a VM, not even its own server, but just a container, right? You would need more than 40 times the current CPU production in an, in a calendar year in order to suppor…”
Prince: Most companies and content creators want their content included in AI
“The majority of companies that are out there, the majority of content that's created They actually want that content in AI, like Cloudflare, like our knowledge base is a way of us sharing how to use Cloudflare really well. And we want to make sure that every A…”
Prince: Web Crawls Per Publisher Referral Click Doubled Over Past Year
“Everything has gotten basically, assuming you're a publisher, from the publisher's perspective it's gotten twice as bad. They just double every one of those numbers over the last year.”
Prince: Cloudflare Handles 500M Transactions Per Second vs Visa's 20,000
“Visa, largest payments network in the world, handles at peak during the holidays, somewhere around 20,000 financial transactions per second. Cloudflare handles about half a billion internet transactions per second.”
Prince: AI micropayments must scale beyond 100 million transactions per second
“We think that somewhere between one and 10% of those will day one be monetizable through agents with a with a micropayments transaction, which means that if you're going to build something like this, you've got to build a payments infrastructure that can day o…”
Prince: Google Crawls 2x More Web Content Than OpenAI
“Google, to give you some sense of these, more than two times as much of the web as open AI More than four times as much as Anthropic, more than five times as much as Microsoft.”
Prince: Cloudflare will block Google scrapers for publishers starting in September
“And what we have said at cloudflare is that by mid September, we're gonna treat Google the same way on. If you have an, you know, ads, if you're a business that's ad supported, subscription supported, then we're going to restrict Google from being able to acce…”
Prince: 2026 Will Be the Year Media Creators Get Paid by AI
“I really think that 2026 is going to be the year that in the media space, in the content space, that you're going to see a real shift and then content creators are going to get paid.”
Prince: Google Paying Publishers for AI Scraping Is Inevitable
“I don't know if they'll pay up before the deadline, but I think it's inevitable that they'll pay up. Because again, I think that they need the content and they understand fundamentally that journalists and content creators and academics deserve to eat.”
Prince: AI Purchasing Agents Will Systematically Favor Incumbent Mega-Corporations
“The agents have infinite time to research every possible detail. And so they're going to look for where the richest sources of information that are online, which are going to be for the established big companies that are out there.”
Prince: Cloudflare is collaborating with Shopify and payment giants on AI commerce
“I'm spending a lot of time with, you know, Shopify you know, PayPal, Visa, MasterCard, American Express you know, Intuit, the companies that actually deal with these small businesses and saying, how can we bring together the tools in order to make sure that th…”
Prince: Ad-free AI agents will widen the global wealth divide
“And then by the way, we get to worry about the next problem, which is that AI is this massive divider between, you know, the rich and the poor. And if we're not in an ad based business model in anymore, which I think is likely if you're spending thousands of d…”