The Exchanges

Q Yeah, ok, sounds good. Just wanted to ask that, I was curious about it, but let's talk, ok now, so you're gonna build a, a technological solution that will block crawling. Yes. And so robots.txt, which is this code that you put in, like, the header of your, of your site, if you don't want to be crawled, that wasn't working?

A Yeah. I mean, I think robots.txt has two problems. Um, the first is some people just ignore it. Uh, and so if you ignore it, then you can still crawl all you want. And there's some just, there's some, even some big legitimate companies that completely ignore robots.txt. And we're really good at basically being able to say, okay, here's what robots.txt says. How are you, are you actually following what those, those, uh, what, what sort of the rules of the road are? And if the answer is yes, then robots.txt is a great solution. Um, but in the cases where somebody is ignoring it, then we need to actually put in place additional technical barriers to restrict their, their, their access. And so that's exactly what we're doing. The second problem with robots.txt is it's not granular enough. So take the Google bot, for example, Google's crawler does five different things, at least. Uh, one is it checks if you have an ad on a page. It makes sure that if you're putting an ad for Procter & Gamble, Procter & Gamble product up, it's not against a pornographic site or something like that. So it does brand safety checks. Um, the second thing that it does is crawls to index for traditional search, the 10 blue links that are out there. The third is that it crawls to create answers that are in the answer box. The fourth is that it crawls to create answers that are in the AI overview, the newer …

AI assessment note: “I think robots.txt has two problems. Um, the first is some people just ignore it.”

Q from the standpoint of a consumer, could this be good? I mean, it's pretty annoying to, uh, type this question into Google. When was Cloudfair, uh, uh, founded? And then have to click to Cloudflare's website to get the answer that Google could just service for you, and so much of the web has sort of become effectively the service of Google queries, where websites don't really need to exist.

A Well, you know, so absolutely this has been happening for a while, and if you look at up until six months ago, the ratio 10 years ago of crawls from Google to clicks was two crawls, one click. Six months ago, it was up to six crawls one click, and that's all because of the answer box. Um, what the AI overviews, which they've rolled out over that time, have done is they've taken it now to 18 crawls to one click. So yes, it, it is a situation of, you know, the frog boiling in, in water, but that's, it has gotten progressively worse. And I think across the media industry, it's gotten harder and harder to actually survive as a, as a publisher. And so what I worry about is, Yeah, you know, the publishers are struggling at, they were struggling at six to one. They're struggling at 18 to one. I think they're dead at 250 or 1500 to one that we're seeing with OpenAI and completely dead at 60,000 to one that we're seeing with something like Anthropic. And so that is the direction that things are going, and that's a challenge. I think you're exactly right on the other point as well, that the, that the challenge here is that this is actually a better user experience. That's why more of the web is going to turn to AI. Um, it is great that you can type something in and you can get back an actual response as opposed to having to hunt for it yourself. That's a better user interface. And so tha…

AI assessment note: “this is actually a better user experience. That's why more of the web is going”

Q How many of your clients, like, do you think will actually go ahead with, um, the blocking of these AI crawlers? I'm talking about not, not your whole client base, but just the news publishers.

A Just the, just the folks who are ad supported. So again, we've done this that like, if you're, if you're not ad supported, none of this applies to you. Like you can, you can set. But if you're ad supported by default, starting mid September, we're going to, you know, switch this over and you can, and we can opt out, but, but we expect that the vast, vast majority of people that have ad ad or subscription supported. Uh, businesses will, will do this. So there will be millions of sites that will drop off of Google's radar. And at the same time that we will, um, very much, uh, tighten up all of the other controls to on, on AI crawlers. So right now it's basically put sort of a no trespassing sign in the top line and say, don't, don't come and get this content. And most people are actually pretty good at respecting the, the no trespassing sign. Starting in mid September, we're gonna actually make it not just as no trespassing sign, but we're actually gonna have, basically put a bouncer there saying, Unless you're compensating creators, you don't get this content.

AI assessment note: “we expect that the vast, vast majority of people that have ad ad or subscription supported”

Q Could there be some collateral damage, uh, with the solution like you're implementing? For instance, I'm looking at the names of these publications, Condé Nast Time, the AP, the Atlantic. I imagine they get a lot of traffic from search as it is today. So if you put this blocker up, does that impact their SEO for instance?

A Yeah. So we've been very, very careful to say that the traditional search today is not blocked. Um, and, and even AI driven search today isn't blocked, but you're going to see us give publishers the tools to differentiate between search indexing, uh, and derivative content. So the way I would think about this is the Google experience today. It may be that publishes. I still want to appear in the 10 blue links, but I don't want to be in the AI overview or the answer box and the granularity of being able to say, okay, Google, I understand you use one bot, but we need that to be treated similarly. And, and again, I am hopeful and in my conversations with Google, um, I, I am increasingly hopeful that they understand the importance of this and giving that granularity. But if for some reason they don't, I am also a hundred percent certain that regulators are paying a ton of attention to this. And that around the world, you will see them force Google to split their crawler out into, into announcing exactly what it is doing. Again, I, I, I think that that's kind of the, hopefully we get to an agreement with Google way before that has to happen. Um, but, but that's, uh, inevitably I think Google is going to have to say, you know, if you don't want us to use your content for derivatives, you have a way of, of, of controlling that. While still appearing in search.

AI assessment note: “we've been very, very careful to say that the traditional search today is not blocked”

Q Okay. I'm going to email you after this. And then when it comes to cybersecurity, obviously you talked about how you're dealing with all these governments that would like to hack into sites across the web. Have they been able to use generative AI tools or automated coding to get more, to become more effective at what they do?

A Yeah. I mean, I think that, that anytime a new technology comes out, bad guys are going to use it as well as, as good guys. And so we have seen, and we will continue to see some horror stories around, you know, the family that, uh, was tricked by some gang in order to wiring their life savings because something, someone that sounded like their, Their daughter called and said, I've been arrested in Mexico. You know, I need, I need to pay to get out, um, or, or, or other things. I think we were seeing a real rise in, especially out of North Korea, North Koreans posing as if they were, um, applicants, uh, to various, uh, various jobs. And then that is, um, you know, allowing them access, which they can then use to, um, to do, to do any number of nefarious things. All of that again, assisted by AI. So I think that's, um, that's been, that's been sort of on the, on the bad guy side. The, the good news though, is that the good guys, you know, folks like Cloudflare, we have been using AI as well in order to not only detect these things, but get smarter at detecting attacks earlier in the process.

AI assessment note: “All of that again, assisted by AI.”

Q very happy, very clearly is not happy with the New York Times. Uh, pursuing OpenAI and especially the actions that the Times are taking in their lawsuit, like forcing OpenAI to preserve their chat logs, which I think is wrong. Uh, but, but it is interesting. So what do you think about is there, are we going to see an evolution from these one-off deals to this marketplace style world?

A Well, I think that, I mean, we've seen this story many times before. I mean, Napster was along, it was a wild west. There were a bunch of lawsuits from, you know, the publishing, the music industry. Uh, targeting Napster and the like, and then along comes iTunes, which starts out as 99 cents a song, but eventually evolves into what is much more, much closer to a Spotify model of a subscription and a pool of funds that then get distributed out to all the creators. So, so I think we've seen this story before. Um, and I think that one of the things that's really important is that open AI and others are willing to pay for content. They do the deals that are there. And I don't think it's right to just say, we'll do a deal to avoid The void lawsuits. Again, I think that, that when you talk to leading AI companies, they understand that people are doing the work to get, create content. They need to get compensated for that content. And if it's not gonna be through subscriptions or ads or ego, it's gotta be through something else. And so exactly how that happens, we'll figure out. But what I know won't work is if open AI is paying for your content, but you're giving it away for free to everyone else.

AI assessment note: “eventually evolves into what is much more, much closer to a Spotify model”

Q first of all, to anticipate a 1000 X increase in web traffic coming from bots in five years. I mean, yes, like, you know, we we've seen, uh, or you've seen quite an increase recently, but in a thousand X is, is a crazy, uh, uh, number. So what is going to make up that thousand X? Like we're, what makes you believe that that's going to be the case?

A I mean, we just look at the trends right now. Um, I mean, in, in May, we saw that, that, um, you know, the automated traffic crossed and we kind of extrapolate out what that rate of change is. And then we assume that again, the, the, the exponent is going to, is going to increase, uh, on, on, on that. Cause, cause it's, you know, it's still way under penetrated, uh, with what's there, but we've got to invent New technologies in order to support it. So just to give you some, some sense. Let's imagine that just every knowledge worker on earth had an agent that was working for them, 24 seven, right? Helping them do their job. And I, and again, that's, that seems reasonable. Like every knowledge worker is going to have some agent that's, that's working for them. Very few do today, really. But, but over time, like that, that becomes, becomes reasonable. Just with that, And putting GPUs aside, just looking at CPUs. If every one of those agents ran in a traditional container, like, you know, which is, which is not even a VM, not even its own server, but just a container, right? Um, you would need more than 40 times the current CPU production in an, in a calendar year in order to support that level. And that's with one agent. I would guess that most knowledge workers are going to have many agents that are going to be working on their behalf. So we just don't, we using the current techn…

AI assessment note: “Let's imagine that just every knowledge worker on earth had an agent that was working”

Q Um, and let's just at the very beginning kind of talk about like the threats that you see to websites, who's trying to take them down. Yeah. What, what's happening on that?

A Yeah, so, I mean, we, protecting websites is, is part of our business. So is, um, protecting employees as they go out across the internet. So we, we, we, Cloudflare is fundamentally kind of a network that is built with all the performance, reliability, security, availability, and privacy guarantees that frankly the internet should have been built with had we all known what it was going to become. But, but obviously back in the sixties, seventies and eighties, when we were laying down all these protocols, We didn't think about those things. And so Cloudflare is basically reverse engineering, uh, the internet in order to give it those performance availability, uh, security, reliability, and, um, uh, and, and, uh, and privacy guarantees on top of, of what, what is there. And so today, one of the main uses for Cloudflare would be to, you have, you're putting a website or a web application or anything online, You want to make sure that it's safe from different sorts of threats, and so what are the threats that we see? I mean, every day we go to war with the Chinese government, the Russian government, the North Koreans, I mean, everyone is trying to hack into our customers, because who are our customers? Some of the largest banks in the world, some of the largest governments in the world, and they are all constantly under threat and constantly under attack from these, these organizat…

AI assessment note: “every day we go to war with the Chinese government, the Russian government, the North Koreans”

Q if there, if the ability to create content is constrained by a human's ability to create content, then you have something to bring to these AI companies. But if, Human plus bot content starts to become the norm. There's going to be so much, then even if you're creating high quality stuff, it's not going to matter very much to these generative AI companies. What do you think about that?

A I, I think it's still, I mean, so first of all, I think that there's the pure AI generated content. There's lots of research that shows that training AI on AI data, uh, is sort of like that old Michael Keaton film, Multiplicity, where basically every copy of something gets worse and worse and worse. And again, that, that feels like that's going to still be, still be the case for quite some time. May in the future, um, robots be able to go out and do, you know, interesting reporting from the field. May they be able to do, you know, interesting research for sure. But today, um, I, I think that that interesting research, that interesting original content, that interesting insight that, that comes from the work that right now only journalists and researchers and others can do is still the most important thing for filling in those gaps in the, in the Swiss cheese of, of AIs. What is just, again, high volume, low value content? My hunch is that that's, if we score it correctly, going to be exactly what it, what it is, which is low value content. And so it should, it should be rewarded very minimally. Um, I, I, I, I like to ski. So I, I live part of the year in Park City, Utah.

AI assessment note: “interesting original content, that interesting insight... is still the most important thing”