robots.txt, every mention
1 scene · ← back to robots.txt
tap a year for its mentions
every year anyone Joel de la Garza 3
Verbatim, from the transcripts: the passages where robots.txt comes up
AI is Revolutionizing Web Security - Bots, Agents, & Real-Time Defense
- ▶ 5:11 Joel de la Garza Well, and I believe we had sort of standards in the old days that developed or quasi standards like robots.txt, right, which would tell you like and tell the crawlers, hey, don't crawl these directories. 3 times in the scene