AI Crawlers
topic on 5 shows · 6 statements across 5 episodes
Latent Space 
the Startup Ideas Podcast 
the a16z Podcast 
Big Technology 
TBPN
6 statements about AI Crawlers, every show
Cloudflare announces hard-blocking AI crawlers unless they pay creators
“And then just today on July first, we announced that we are hard blocking the AI crawlers unless they will actually compensate Content creators for the content that they're creating.”
Prince: AI crawling for training and RAG imposes bandwidth costs on publishers
“I think in both cases though, you're actually costing the content creator something. There is literally, they're paying for that traffic. They're paying for that load that the crawlers are Pulling off of it.”
Isenberg: Cloudflare Launched Pay-Per-Crawl Feature for AI Scraping
“Cloudflare. They just launched pay per crawl. Websites can now charge AI crawlers for scraping content.”
McCloy: Allowing AI crawlers to train on short-lived promotional data is counterproductive
“You can think about the half-life of the information you're presenting and whether or not that's something that really makes sense to give to a training data crawler. So for example, like, you know, if you're an e-commerce company and you're running a seasonal…”
Cloudflare enables free default blocking of uncompensated AI web crawlers
“Starting on July first we flipped the script. We made it as a default that if you're using Cloudflare that you can by default for free block any crawlers from taking your content without compensating you.”
Mytton: AI crawlers will increasingly adhere to web rules within 18 months
“And so over the next 18 months, I think we'll see more of that, more of the AI crawlers that we want, following the rules, doing things in the right way. And it will start to split into making it a lot easier to detect the bots with criminal intent, and those …”