Blocking AI training in Cloudflare can also shut out Googlebot

Cloudflare deployed the reworked system on September 15, 2026, according to a dev.to write-up of the change ^[2]. For new domains, the default lets Search through. On pages that carry ads, the default shuts out both Training and Agent crawlers ^[6].

A crawler that does both jobs sits in two categories at once. The write-up says that when Training is blocked, the combined behavior can also block a crawler such as Googlebot ^[7]. It does not disclose which default wins for Googlebot on an ad-supported page, the case the author flags as most exposed ^[13].

The old model had one switch, "Block AI Bots" ^[1]. The new one, a better model in our view, classifies traffic by purpose: Search for discovery and indexing, Agent for AI agents completing tasks or retrieving information, Training for collection used to train models ^[3][4][5].

There are two documented ways to keep indexing while refusing training. Cloudflare provides a Disallow AI Training option, which the write-up describes as the Cloudflare path built to preserve indexing while preventing training ^[8]. The write-up also identifies tokens such as Google-Extended as another route ^[9]. Which one fits depends on the site's goals and its Cloudflare configuration ^[9].

Some sites will choose the broad block on purpose. The write-up notes that a site restricting all automated access may accept reduced search availability ^[10]. The author's advice is a focused review, not an automatic switch to the most restrictive setting ^[15]. The sequence it lays out:

1. Identify whether the site uses Cloudflare's AI traffic controls and whether it carries a legacy configuration ^[11]. 2. Establish which access matters most: search indexing, agent visits, training, or a combination ^[12]. 3. Review the chosen controls against their effect on multi-purpose crawlers ^[13]. 4. Verify the intended crawler access after any change, instead of assuming a broad block touches only training traffic ^[14].

In our view step four is the one that settles it. A setting labelled Training is only confirmed as a training setting when Googlebot still reaches the page afterward.

The account is secondhand. dev.to summarizes Cloudflare's announcement and says the outcome depends on Cloudflare's settings, a site's existing configuration and the crawler's combined behavior ^[16]. Two sites with the same Training setting can therefore see different results for Googlebot.