Jul 23, 2025 · 56m · latent-space
AI is Eating Search
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space Podcast, Scrunch.ai co-founder Robert joins Alessio and Swix to break down the mechanics of AI search engines, the transition from traditional SEO to Agent Experience (AX), and technical strategies for capturing high-intent commercial traffic.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 3.2 Guest teaching 5.3 Guest disagreement 1.8 The hosts pushing back 1.6
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 45:51 Robert rejects the need for iterative link navigation in deep research
Robert directly counters Swix's suggestion of manipulating deep research trajectories with follow-up links, arguing it is far better to simply provide complete answers on a single page.
Hardest push from the hosts ▶ 3:20 Swix challenges Robert's anti-chatbot generalizationSwix pushes back against Robert's claim that site visitors hate chatbots by pointing to Intercom's massive commercial success with regular consumers.
Biggest teaching moment ▶ 17:35 Robert explains the LLM re-ranking mechanism and metadata penaltyRobert educates the hosts on the two-stage retrieval pipeline, demonstrating why human-targeted clickbait metadata gets systematically downranked by LLM search agents.
The host holds their own ▶ 11:51 Swix articulates the free distribution philosophy for AI indexationSwix demonstrates deep industry perspective by breaking down the ideological divide between paywalled publisher models and zero-marginal-cost open distribution for AI authority.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Origins of Scrunch.ai and the Shift in Web Discovery | 2 | 4 | 2 | 2 | Alessio and Swix open by exploring the genesis of Scrunch.ai. Swix gently pushes back against Robert's claim that nobody wants website chatbots by pointing to Intercom's commercial success, prompting Robert to clarify the nuance between customer support rapport and cold discovery. | |
| Monitoring AI Platforms and Managing Pre-Training Crawler Exposure | 3 | 5 | 1 | 1 | Alessio queries how Scrunch tracks both search-enabled and pure pre-trained model outputs. Robert educates the hosts on managing crawler exposure based on information half-life, such as blocking temporary e-commerce sales from permanent training weights. | |
| Publisher Paywalls and AI Replacing Traditional Web Browsing | 4 | 5 | 2 | 2 | Swix introduces Cloudflare's bot paywalls and asserts the philosophy of making content free for LLM training. Robert offers a broader perspective, explaining that AI is not just changing search engines but fundamentally replacing traditional web browsing itself. | |
| Mechanics of AI Search, Ranking, and Metadata Optimization | 4 | 7 | 1 | 1 | Alessio shares traffic statistics from Lenny's newsletter and asks how ranking mechanics work under the hood. Robert breaks down the two-tier retrieval architecture, noting that traditional indexes retrieve candidates but LLMs re-rank them based on informational metadata rather than clickbait. | |
| Search Intent Tracking and Hallucination Signals | 3 | 5 | 1 | 1 | Alessio asks whether site operators can inspect incoming AI prompts to dynamically rewrite landing pages. Robert explains privacy boundaries prevent query-passing while demonstrating how indirect traffic landing patterns signal unserved user intent. | |
| Moving Beyond SEO to Agent Experience (AX) | 3 | 5 | 2 | 1 | Alessio asks about industry taxonomy across AEO and GEO. Robert clarifies that positioning should move beyond entry-point discovery toward Agent Experience (AX), managing end-to-end multi-turn interactions across AI agents. | |
| Personalization and Intent Clusters in AI Queries | 3 | 6 | 3 | 2 | Swix and Alessio probe into prompt clustering and user intent data. Robert disputes mainstream clickstream studies that claim Google is safe, arguing that problem-solving intent and B2B software bake-offs in AI search capture higher commercial value than classic search. | |
| Prompt Injection Realities and the Limits of llms.txt | 3 | 6 | 2 | 2 | Swix asks about using prompt injection to manipulate search rankings. Robert cautions that short-term black-hat exploits will face platform penalties, while also debunking the widespread assumption that ChatGPT actively indexes llms.txt for discovery. | |
| Technical SEO Pitfalls: JavaScript Crawling and Programmatic Content | 4 | 5 | 2 | 2 | Swix and Robert discuss technical implementation flaws, highlighting that AI crawlers fail on client-side JavaScript. Robert then outlines how programmatic SEO can be effective when transforming internal support tickets rather than scraping public web content. | |
| Deep Research Mechanics and Content Consistency | 3 | 6 | 3 | 2 | Swix suggests optimizing for deep research by embedding suggested follow-up links to control exploration branches. Robert rejects the premise, pointing out that brands should provide comprehensive single-source answers rather than forcing iterative lookups. | |
| Real-World Case Studies: Clerk's Traffic and Conversion Lift | 3 | 5 | 1 | 1 | Alessio asks for concrete case studies and questions how much of the work is tooling versus services. Robert shares Clerk's 6x AI traffic and 9x conversion growth, explaining Scrunch's role as an experimentation platform rather than a creative agency. | |
| AI Search Market Share Dynamics and Hiring Callout | 3 | 5 | 1 | 2 | Swix prompts Robert on AI search market share dynamics across Perplexity, Claude, and Meta AI. Robert analyzes ChatGPT's dominant consumer retention and high-intent niche audiences in technical and financial demographics. |
Statements from this episode (25)
McCloy: Users dislike on-site website chatbot popups
“Nobody wants to use a chatbot on your website. I don't want to use chatbots on people's websites. Like if things pop up on a website when I go to visit it, I just like, I fly into a visceral rage. I'm definitely closing the window if not closing the tab.”
McCloy: ChatGPT user experience is vastly superior to Google search
“The experience people have of using ChatGPT versus, like, Browsing the internet, sort of the old Google 10 blue links and clicking around is like vastly more positive, you know, compared to the past.”
McCloy: Claude users represent an exceptionally valuable audience for companies
“Claude, which is important for, you know, not necessarily huge in terms of raw number of users, but the people who do use Claude tend to be like a very valuable audience, especially for some types of company.”
McCloy: Meta AI is a 'silent monster' driven by distribution reach
“Meta AI, for example, which I don't think gets necessarily remarked upon a ton in the like AI enthusiast, AI developer community, but like it's a silent monster, right? Because of Meta's distribution and reach.”
McCloy: Most high-intent commercial AI queries increasingly rely on live search
“Most of the queries that lead to action that are sort of commercial interest are increasingly using search across these systems, like for a bunch of reasons, right? Timeliness and avoiding hallucinations and so on.”
McCloy: Allowing AI crawlers to train on short-lived promotional data is counterproductive
“You can think about the half-life of the information you're presenting and whether or not that's something that really makes sense to give to a training data crawler. So for example, like, you know, if you're an e-commerce company and you're running a seasonal…”
McCloy: Stopping LLMs broadly from consuming web content will be nearly impossible
“You can fight the battle, I think, of saying, OpenAI shouldn't consume your content without paying for it, or shouldn't consume it at all. But I think it's gonna be really tough to fight the battle of saying, like, LLMs writ large shouldn't consume my content,…”
McCloy: AI is fundamentally replacing web browsing, not just search
“Thinking about what's going on in AI as really replacing search is the wrong take, or maybe not that interesting. It's really more like AI is replacing web browsing. I think that's actually like a more meaningful, like more fundamental shift.”
McCloy: AI search platforms still rely on traditional search algorithms under the hood
“Most of the search that's in AI search is still traditional search, right? There's an AI model in front of it, but you're still using your sort of traditional, like, you know, text-based, like, BM-Five or TF-IDF or PageRank or what have you search algorithms u…”
McCloy: ChatGPT filters out clickbait metadata in favor of descriptive information
“The metadata has been super optimized for human click through, right? So think about like clickbait, think about things that are like, you know, like Creating urgency, especially true in e-commerce, right around sales and things like that. But they're not very…”
McCloy: ChatGPT will not pass user prompts to destination websites
“The reason I think it probably won't be the case in the future is for privacy reasons. So I don't think that you will see, at least in the context of the way search is working today inside like a system like ChatWD, I don't think you'll see that it's going to …”
McCloy: Gray-Hat SEO Tactics Currently Work in AI Search
“In terms of like some of the more gray area SEO tactics like that, you know, Penguin was addressing with Google back in the day. I mean, I think the reality is a lot of those things do work today. Like I can't tell you they don't work, but I would say as like,…”
McCloy: AI Search Products Lack Google's Robust Abuse Detection
“None of these products have as much Attention to detail or engineering put into them in terms of, like, moderation and abuse detection and, like, being defensive against the fact that the Internet's a wild place, as Google does, right?”
McCloy: Optimizing for AI Agents Is Essential to Serving End Users
“I think it's going to be really important to Be that data-driven about how these, like, AI systems are interacting with your content, with your website, with your infrastructure, because ultimately they are serving your ultimate customer, serving the user, and…”
McCloy: ChatGPT personalization and custom preferences directly alter AI search retrieval and sources
“ChatGPT personalization, memories, just explicit preferences, if you set them up, do definitely affect the results you get. Now, Again, obviously, there's, they're still using traditional search, so the search index itself is not necessarily personalized, but …”
McCloy: AI chat queries represent higher commercial intent than Google informational searches
“In chat search people, people don't really do that. People just say, solve the problem for me. What I think is really interesting about that is, like, it's actually the highest intent. Like, it's the most valuable thing you could be doing. Because you're going…”
McCloy: Enterprise users are outsourcing B2B software vendor bake-offs to ChatGPT
“One specific example that takes people a little bit by surprise is, like, there are a huge number of people doing, like, B to B software bake-offs in chat to VT. You know, if you're at a big company and your boss is like, Hey, we need to buy something to solve…”
McCloy: ChatGPT Search Bans for Prompt Injection Are Coming
“I think it works until it stops working.
Right.
And I would say like, there's not a lot of stories of people getting banned for like Chatsby D search so far, but it's coming.”
McCloy: ChatGPT Does Not Index or Retrieve llms.txt by Default
“I knew that there's debate about this, but I'd say the evidence is like ChatTriPT is not indexing and it's not retrieving content from LMS.txt by default.”
McCloy: Optimizing for semantic embeddings does not improve AI search ranking
“But I think as a tool to understand how these AI platforms are consuming your content, embeddings aren't that relevant. You're better off focusing on, like, just, again, having a good structured set of content that makes sense, like, to a human with, like, cle…”
McCloy: ChatGPT and most AI search indexers do not execute JavaScript
“Chatsby doesn't execute JavaScript. Most of the AI Search indexers and retrievers don't execute JavaScript. So if your content is not rendered on the server side it's typically not going to be available.”
McCloy: Clerk Achieved 6x AI Traffic Growth and 9x Lift in Conversions
“So I think the stat is like, they've seen like a six X growth in ChattoBD traffic or traffic from the ChattoBD propensity, but the common set, but they've seen a nine X lifted conversions.”
McCloy: Active AI Search Optimization Delivers Double-Digit Traffic Gains in 1–2 Months
“So I would say we typically see people get double digits improvements in traffic within a month or two if they, you know, if they are actively working.”
McCloy: ChatGPT holds the most durable consumer presence among AI platforms
“ChatGPT, I would say, by far, in a way, is the thing that people think about as being an AI platform that has the strongest consumer presence and the most durable relationship with consumers.
Right.
So everything else is kind of a distant second.”