Product1 publisher3 min readPublished
Cloudflare lets site owners admit only the AI bots that might pay
Cloudflare says bots are over half of internet traffic and lets sites block AI tools, allow them, or admit only those that might pay. With payment schemes still unsettled, picking one is a revenue bet made on the gatekeeper's own numbers.
The Product Desk · Product desk
What happened
- Cloudflare found in June that bots made up more than half of internet traffic, a share the Verge says keeps rising.
- Cloudflare sits between websites and AI tools, and site owners can block those tools, allow them, or allow only the ones that might pay for access.
- The Verge describes the payment schemes for AI access as still coming into focus, including which kinds of information will turn out to be valuable.
- Earlier this year Cloudflare laid off more than a thousand people, 20 percent of the company.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- decision The bot setting now decides whether AI products can use a site's work and on what terms, so the choice belongs with whoever answers for revenue, with the hosting team advising.
- constraint Choosing paid access today holds content back while buyers and prices are unsettled, so in the near term the paid option behaves much like a block.
- exposure A site that relies on Cloudflare both to size its bot problem and to charge for access depends on one company for the measurement and for the gate.
Somewhere this week, the person who owns a publisher's revenue line will log into a Cloudflare account that used to be the hosting team's and find a bot setting with a paid option in it [2].
On his previous Decoder appearance, Prince described Cloudflare as "a service that makes the internet faster and protects it from bad guys" [6]. This time he said that is what he says "when I'm trying to answer a question at a cocktail party and I don't want to talk to the person any more" [12]. The longer version is that Cloudflare is "trying to rebuild the internet the way it should have been built from the beginning if we knew how important it was going to be" [11].
Teams tend to count every visit as a reader. More of the visits now come from software sent on a person's behalf. The Verge describes AI companies scraping more of the web and more AI agents trying to do things for people on it [3]. The bot share is Cloudflare's own internet-wide measurement [1]. Any single site's mix will differ. A traffic report that lumps people and machines together overstates the audience an advertiser thinks it is paying to reach.
The pitch is large. Prince said Cloudflare is "trying to help shape what that future business model of the internet looks like and make it as healthy as possible" [5]. The product is a setting with three positions, and only the paid one brings in money [2]. The interview excerpt does not include a price, a revenue split, or a count of sites that have chosen to charge.
Cloudflare is physically placed to run a gate. Prince said the network is in "over 350 cities worldwide, literally thousands of data centers" [8]. By his own account, though, the growth is elsewhere. He called the platform for building applications on that infrastructure "the fastest-growing part of our business" [9].
Cloudflare is also replacing some of its own people with AI. After the layoffs, Prince wrote a Wall Street Journal op-ed headlined "How I choose which Cloudflare employees to replace with AI" [10].
For the person holding the setting, two axes sort the choice. The first is where the money arrives. It can come when a person lands on a page, through ads, subscriptions or leads. Or it can come when a task finishes, such as a booking or a purchase. The second axis is whether an AI company could get the same material somewhere else.
Task revenue with ordinary content argues for allowing agents, since blocking one turns away a customer who sent software instead of a browser. Page revenue with ordinary content gains little from a block, and charging has no obvious buyer. Page revenue with material only you publish is the case for the paid option, with a block as the fallback until a buyer and a price show up. Task revenue with distinctive material is the one box where admitting shopping agents and charging scrapers both make sense, provided the controls can tell the two apart.
Whichever box you land in, judge the change on human referrals and subscriber retention. Across the web, total traffic is more than half machines by Cloudflare's count [1].
What to watch
- Published prices or revenue splits for paid AI access through Cloudflare, and the first AI companies that agree to pay them.
- A count of sites choosing the paid option versus blocking AI tools outright.
- An independent measure of bot share that confirms or contradicts Cloudflare's more-than-half figure.