Cloudflare splits search from training by publishing a no-training line in your robots.txt

The no-training part of the deal takes effect after the request is served. An Accountable crawler still gets the page; the preference governs what its operator does with the bytes afterwards. Cloudflare says its approach depends on crawler operators accurately declaring and honoring the relevant use of collected content ^[9].

That dependency is structural. A fetch for /pricing looks the same whether the response ends up in a search index or a training set, and the split between those uses happens on the crawler's side of the wire. The directive Cloudflare writes for you is a request.

For the setting to give a publisher what it means by protection, Cloudflare's classification has to be right about which crawler is Accountable ^[4]. The operator has to declare its use of the content honestly ^[9]. And inside its own pipeline, the operator has to keep search collection and training collection separate. Only the first of those is checkable from outside a vendor.

For two of the three operators Cloudflare says have committed, this is less new than it sounds. Google and Apple already offer mechanisms, including robot directives, for excluding content from AI training ^[11]. Cloudflare lists Bingbot as expected to follow in early 2027 ^[8], so of the three commitments, two have a crawler on the Accountable list today and one has a date ^[17].

The training signal is one of three. Cloudflare also classifies Search behavior and Agent behavior, the latter covering human-directed or bot-assisted access such as chat retrieval bots ^[5]. Disallow AI Training sets a training-specific preference and leaves the other two categories to their own settings.

The thing I would check first on a new domain is the preset. Cloudflare gives new domains one of two onboarding presets, chosen by whether advertising monetization is involved ^[13]. The dev.to account of the announcement does not say what either preset configures ^[18], so the starting policy on a fresh domain is whichever branch Cloudflare picked for it.

There is also a second author on robots.txt now. A team that keeps a hand-written robots.txt in version control will be serving a file whose contents come partly from a dashboard setting. Cloudflare plans to move away from Managed Robots.txt toward Bot Preference Sync ^[14].