Cloudflare will block AI crawlers by default from Sept. 15 after bots overtook human traffic

For an ad-funded page, Google's crawl ratio says more about revenue than the headcount crossover does. At about six crawls per referred visitor, 600 hits from Google's crawler sent back roughly 100 readers. At 18:1 the same 600 hits send back about 33, a two-thirds fall in readers per crawl ^[3][1]. Crypto Briefing puts the current figure at "18:1 or worse" ^[3], so 33 is the generous case. None of those crawls earn ad revenue ^[9].

Prince's 2031 forecast ^[2] starts from something close to parity in May 2026 ^[1]. Getting from one to 1,000 in five years requires the bot-to-human ratio to grow about fourfold every year, since four to the fifth power is 1,024 ^[2]. The forecast comes from the chief executive of a company that routes and protects a substantial portion of the web's traffic ^[7]. The May crossover was measured on that network, and the account of it comes from Prince alone ^[1].

The mix inside AI crawling has changed too. Training crawlers, the ones scraping content to feed language models, account for 40 to 52% of AI crawler activity, according to the report, a change from an earlier wave dominated by search-adjacent indexing ^[4]. That leaves 48 to 60% doing something other than training ^[3]. Prince attributes his 1,000-fold forecast to AI agents running tasks far more often than any person browses ^[2], so the growth he expects sits mostly outside the training share.

The default can go three ways from here. AI companies accept it and pay for access, per crawl or by micropayment, the model Prince has pointed toward ^[6]. Publishers opt out in large numbers to keep AI agents reading their pages, and the block changes little ^[5]. Or AI companies work around it, licensing directly from the biggest publishers or crawling the part of the web Cloudflare does not carry ^[7]. The report does not include a per-crawl price.

I'd expect the block to hold on most small sites, because a default decides for anyone who does nothing. Those are the sites that, according to Crypto Briefing, could not identify and block crawlers on their own ^[10]. The counter-thesis is about where the money sits. The largest publishers, whose pages AI companies most want, could already block and negotiate for themselves (the report's point about small and mid-sized sites implies as much ^[10]), so the default adds the least leverage where payments would be largest. If per-crawl fees end up reaching mostly small sites, ad-funded publishers are still serving 18 crawls or more for every reader Google sends ^[3].