McCloy: ChatGPT Does Not Index or Retrieve llms.txt by Default
Robert McCloy · AI is Eating Search · Jul 23, 2025 · at 38:52
Robert McCloy, co-founder of Scrunch AI, explains the current limitations of llms.txt files for AI search engine discovery.
0:00 / 0:07exact quote · 7.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I knew that there's debate about this, but I'd say the evidence is like ChatTriPT is not indexing and it's not retrieving content from LMS.txt by default.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Robert McCloy
McCloy: AI is fundamentally replacing web browsing, not just search
“Thinking about what's going on in AI as really replacing search is the wrong take, or maybe not that interesting. It's really more like AI is replacing web browsing. I think that's actually like a more meaningful, like more fundamental shift.”
McCloy: Gray-Hat SEO Tactics Currently Work in AI Search
“In terms of like some of the more gray area SEO tactics like that, you know, Penguin was addressing with Google back in the day. I mean, I think the reality is a lot of those things do work today. Like I can't tell you they don't work, but I would say as like,…”
McCloy: AI chat queries represent higher commercial intent than Google informational searches
“In chat search people, people don't really do that. People just say, solve the problem for me. What I think is really interesting about that is, like, it's actually the highest intent. Like, it's the most valuable thing you could be doing. Because you're going…”
McCloy: ChatGPT Search Bans for Prompt Injection Are Coming
“I think it works until it stops working.
Right.
And I would say like, there's not a lot of stories of people getting banned for like Chatsby D search so far, but it's coming.”
McCloy: Users dislike on-site website chatbot popups
“Nobody wants to use a chatbot on your website. I don't want to use chatbots on people's websites. Like if things pop up on a website when I go to visit it, I just like, I fly into a visceral rage. I'm definitely closing the window if not closing the tab.”