robots.txt
1 statements across 1 episodes · 0 bullish · 1 bearish · 1 people on the record · first statement Aug 2, 2024 by Dave Morin · across every show →
Everything said about robots.txt, oldest first
Aug 2, 2024 negative
Morin: Anthropic rotates bot names to bypass web scraper blocks
“And suddenly you've got this problem that I was reading about this week where Anthropix changing the name of their bots. They're creating 10 more bots. Developers are having to, like, you know, block this bot and that bot and this name and that name. It's a bu…”