Cloudflare's Bot Preference Sync makes robots.txt an output of the edge, not a policy of its own

Cloudflare's stated reason for building this is behavioural rather than tidy-desk. When a site's published preference and its enforced rule disagree, some crawlers treat the disagreement as grounds to disregard the preference or to try bypassing the block, the company writes in its announcement ^[9]. Accept that and robots.txt changes job. It stops being the policy and becomes corroboration for the policy, which lives in the rule that drops the request.

The two layers drifted in the first place because bot identity is not stable. One user agent can build a search index, fetch a page for an AI agent, and collect material for model training, and Cloudflare's July taxonomy treats those as three separate behaviours even when a single operator performs all three ^[15]. Cloudflare's July 1 argument was that these mixed-use crawlers disadvantage site owners precisely because they make the wanted use inseparable from the unwanted one ^[16]. A hand-maintained file cannot keep up with that. A generated one can, at the cost of depending on whoever maintains the generator.

Then there is what "Disallow Training" actually means at the edge. Selecting it makes Cloudflare publish a no-training preference while keeping separate enforcement against crawlers that fail its transparency requirements ^[7]. Those requirements number four, and only one of them describes crawler behaviour: respecting a no-training preference by any mechanism. The other three are disclosure duties owed to the site owner, namely an opt-out from AI summaries, URL-level visibility into which pages were made available for training along with search metrics, and public evidence that declining training does not cost the site its ordinary search results ^[10][18].

None of that reaches a crawler unwilling to say who it is. Cloudflare's verification framework depends on bots identifying themselves and honouring the relevant robots.txt preference, and syncing a file cannot compel an unidentified crawler ^[12]. The company's own reference point for that limit, per the runtimewire account, is the accusation it made against Perplexity last year ^[13]. So this is housekeeping for the cooperative half of the traffic, with the enforcement layer left to handle the rest.

One wrinkle in the reporting: runtimewire describes the feature as rolling out in an upcoming launch ^[19], while Cloudflare's own post announces it as available across tiers and switchable at any time ^[1]. Existing users of the older managed robots.txt feature will be asked to review their settings before moving across ^[3]. New customers get it on by default, which means the published file on a new Free tier site will be Cloudflare's output unless somebody deliberately turns it off ^[20].

That default is the part worth thinking about, because Cloudflare's own examples pull in opposite directions: an e-commerce store that wants everything crawled and trained on so its sofas surface in a chatbot answer, and an ad-funded publisher that wants to stay in the search index while keeping articles out of the training set ^[17]. The file will now say whichever of those the owner clicked. That is an improvement, provided the click was deliberate.