Jun 16, 2025 · 24m · a16z
AI is Revolutionizing Web Security - Bots, Agents, & Real-Time Defense
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the a16z podcast, Arcjet CEO David Mytton and partner Joel de la Garza discuss why legacy network-level blocking of AI traffic fails for modern web applications. They explore granular application-context security, multi-layered fingerprinting, cryptographic proofs, and low-latency edge AI inference needed to manage automated agents effectively.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The host as informed peer 5.3 Guest teaching 4.0 Guest disagreement 0.9 The host pushing back 1.6
speaking balance: gold is the host, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 17:33 Guest rejects assumption that non-human traffic implies malicious intent
In a generally non-combative interview, David mildly challenges conventional security assumptions by emphasizing that treating automated browser traffic as inherently malicious is an outdated paradigm.
Hardest push from the host ▶ 10:28 Host pushes on the reality of controlling traffic at internet scaleJoel presses David on how complex application-layer inspection can realistically function under massive terabit DDoS conditions based on his own real-world scale experience.
Biggest teaching moment ▶ 12:30 Guest explains reverse DNS lookups and TLS fingerprinting mechanicsDavid provides a detailed technical breakdown of how security teams verify bot identities using reverse DNS lookups and client request hashing algorithms.
The host holds their own ▶ 19:19 Host frames identity proofing around NIST working groups and gig-economy historyJoel demonstrates extensive domain authority by contextualizing AI proof-of-humanness within 35 years of NIST identity standardization efforts and gig-economy onboarding challenges.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The host as informed peer | Guest teaching | Guest disagreement | The host pushing back | Why |
|---|---|---|---|---|---|---|
| Title Card: AI Bots at the Edge | 4 | 3 | 1 | 1 | Joel opens by contrasting legacy IP-blocking methods with modern bot nuances, demonstrating familiarity with network security tools. David explains why DDoS solutions handle network traffic while application-layer context is needed for modern AI bots. The tone is highly collaborative. | |
| Modern AI Solutions and Application Context | 5 | 4 | 1 | 2 | Joel asks whether vendor AI claims are just rebranded network telemetry, referencing historical standards like robots.txt. David educates on OpenAI's multi-bot setup and why voluntary standards fail without application-layer enforcement. Joel reinforces the business revenue risks of over-blocking. | |
| Understanding OpenAI Crawlers and Real-Time Query Agents | 4 | 5 | 0 | 1 | David systematically categorizes OpenAI's crawlers, from model training to real-time doc retrieval and search indexing. Joel validates the explanations with concrete user examples like querying JFK's birthday. The dynamic remains entirely conversational and informative. | |
| Computer-Use Agents, Headless Browsers, and Action Control | 6 | 5 | 1 | 2 | Joel shares technical experience handling terabits of traffic across 450k IPs and questions how fine-grained control works at scale. David walks through defense layers, including residential proxies, User-Agent verification, and TLS fingerprinting hashes like JA3 and JA4. | |
| OSI Stack Identity and Header Fingerprinting | 6 | 4 | 1 | 2 | Joel actively maps fingerprinting concepts back to the OSI stack and compares emerging cryptographic pass keys to legacy Kerberos implementations. David details HTTP header order hashing (JA4H) and Apple's Privacy Pass. | |
| AI Agents as Primary Web Consumers and Future Trends | 5 | 3 | 1 | 1 | Joel offers a personal thesis that AI agents will become the main consumers of internet content. David agrees and expands on agent.txt standards and future criminal bot detection models. | |
| Proof of Humanness and Real-Time Edge Inference | 7 | 4 | 1 | 2 | Joel cites the 35-year NIST identity working group and compares inference price drops to historic cloud S3 storage trends. David outlines ultra-fast edge inference models, while Joel closes with an insightful observation about ad-tech applications. |
Statements from this episode (19)
Mytton: DDoS protection has become a commodity handled by cloud providers
“The DDoS problem is still there, but it's just almost handled as a commodity these days. The network provider, your cloud provider, They'll just deal with it. And so when you're deploying an application, most of the time you just don't have to think about it.”
Mytton: Non-DDoS web security requires application-level context
“So a volumetric DDoS attack, you just want to block that at the network. You never want to see that traffic, but everything else needs the context of the application.”
Mytton: E-commerce sites should flag suspicious transactions, not block them
“If you're running an e-commerce operation, an online store, the worst thing you can do is block a transaction because then you've lost the revenue. Usually you want to then flag that order for review.”
Mytton: OpenAI operates four to five distinct bot types
“Particularly with AI coming in where something like OpenAI has four or five different types of bots, and some of them you might want to make a more restrictive decision over, but then others are going to be taking actions on behalf of a user search.”
Mytton: Newer bots use robots.txt to target restricted site areas
“And, but there are newer bots that are ignoring it or even sometimes using it as a way to find the parts of your site that you don't want it to access. And they will just do that anyway.”
Mytton: Websites in AI search indexes gain more traffic and signups
“As we were seeing, sites are getting more signups, they're getting more traffic.”
De la Garza: Traditional bot blocking was driven strictly by scale
“I mean, if I remember back why we made a lot of the decisions we made in blocking bots was strictly because of scale. So, you know, you've got 450,000 IP addresses sending you terabits of traffic through a link that only can do gigabit and you've got to just s…”
Mytton: Residential IP Proxies Render Basic ISP Blocks Ineffective
“And then you have the compounding factor of the abusers will purchase access to proxies which run on residential IP addresses. So you can't easily rely on the fact that it's part of a home ISP block anymore.”
Mytton: Googlebot and OpenAI Bots Can Be Validated via Reverse DNS
“And Googlebot, OpenAI, they tell you who they are, and then you can verify that by doing a reverse DNS lookup on the IP address.”
De la Garza: Major vendors will each launch distinct request-signing standards
“Every, every large vendor is going to have their flavor. And if you're a shop and you're trying to sell to everybody, you've got to kind of work with all of them.”
De la Garza: AI agents will become primary consumers of web content
“And it seems like we're moving to a world where almost the layer you describe, the agent type activity you describe will become the primary consumer of everything on the internet.”
Mytton: 50% of Current Internet Traffic Comes From Automated Bots
“50% of traffic is already bots, it's already automated”
Mytton: Web Traffic From Autonomous AI Agents Will Explode
“We're going to see an explosion in the traffic that's coming from these tools”
Mytton: Blanket Blocking of AI Traffic Is the Wrong Security Strategy
“Just blocking them just because they're AI is the wrong answer.”
Mytton: AI crawlers will increasingly adhere to web rules within 18 months
“And so over the next 18 months, I think we'll see more of that, more of the AI crawlers that we want, following the rules, doing things in the right way. And it will start to split into making it a lot easier to detect the bots with criminal intent, and those …”
De la Garza: NIST identity proofing group has stalled for 35 years
“There's a NIST working group on proofing identity that's been running, I think for 35 years, and like still hasn't really gotten to something that's implementable.”
Mytton: Low-memory edge AI models will be deployed within years
“We're already seeing new edge models designed to be deployed to mobile devices and IOT that use very low amounts of system memory and can provide inference responses within milliseconds. I think those are going to start to be deployed to applications over the …”
De la Garza: ChatGPT is 100% accurate at detecting suspicious emails
“To drop a suspicious email into a, into ChatGPT and ask if it's suspicious, and it's like a hundred percent accurate, right? Like if you want to like find sensitive information, you ask the LLM, is this sensitive information? And it's like a hundred percent ac…”