robots.txt
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Nov 1, 2023 by Aravind Srinivas · across every show →
Everything said about robots.txt, oldest first
Nov 1, 2023 neutral
Srinivas: Perplexity will not crawl websites that block web crawlers
“Yeah, I mean, like, if they don't want to be crawled, we shouldn't crawl them. You know, that's just being how the internet has worked. Like, you know, people have the robots, the text file, and like, you know, if they don't let you do it, then you should just…”