Cloudflare says its traffic signals led OpenAI's crawler to hundreds of millions of new pages

Most of the engineering the post describes is subtraction. Cloudflare says automated traffic can make a nonexistent page look interesting ^[4]. Bots repeatedly ask for files that are not there, some sites redirect almost any address to their homepage, and other addresses lead to copies of the same content ^[5]. From a crawler's side, a redirect to the homepage makes any guessed URL resolve. A discovery system that read raw request counts as interest would send the crawler after all of it.

Cloudflare says recognizing these patterns lets it cut wasted work before a crawler fetches the page ^[6]. It also says the efficiency gain benefits site owners ^[6].

The harmful-page work is the better-designed half. Cloudflare's example is a legitimate site with one compromised page, such as a fake login form planted by an attacker ^[14]. Flagging that page leaves the rest of the domain alone ^[14]. A domain-level flag would penalize every clean page on that site. In our view, per-page flagging is the right granularity for this case.

The count needs more care. Hundreds of millions of new pages is the number of pages added to one index in one pilot, according to Cloudflare, one of the two parties that ran it ^[8]. The cohort was a small slice of traffic, which leaves about 99% of the network outside it ^[7]. Cloudflare's summary of the result is "So far, the answer is a resounding yes" ^[9].

For that count to transfer to a given site, two things would have to be true. The site's traffic would have to look like the cohort's. And its pages would have to be ones OpenAI had not already found.