standard
Root-level plain-text file declaring which paths well-behaved crawlers may fetch; an access policy whose observance is voluntary.
No evidence-backed relationships are recorded.
Cloudflare now sells web search from Ceramic.ai, Exa and Linkup through AI Gateway at partner list prices, billed from gateway credits with no markup. Teams already on the gateway skip a separate search vendor account, though their agents still have to issue each search call.
Reality
Pub-trivia.app's developer warns that Disallow: /dashboard/ would block every dashboard page except the bare /dashboard its homepage footer links to. Blocking a fetch never stops indexing, so the noindex tag has to go live before the Disallow.
Reality
Anthropic told an Australian inquiry a broad copyright exemption is off the table and proposed opt-out 'conditional approval' for AI training instead. The ABC and SBS want the onus on AI firms, so the inquiry is now deciding who pays to police training bots.
Reality
FoxyInvoice prerenders every public Angular route to static HTML at build time so crawlers that skip JavaScript get the full page text. A build step and a one-line drift check take the place of the always-on SSR server its team declined to run.
Reality
Fewer than 1% of Cloudflare sites block search crawlers while 17% block training. A single mixed-use crawler used to collapse those two into one decision, and the new setting splits it for three named operators.
Reality
An ecommerce audit consultant writing in Entrepreneur says most AI crawlers read only a page's initial HTML, so a product page that ranks in Google can return nothing to an assistant fielding around 50 million shopping queries a day.
Reality
Amazon's robots.txt now disallows ChatGPT-User and OAI-SearchBot, and in ZDNET's tests the shopping agent sent a buyer to Walmart, eBay and a retailer called Soaplicity instead of Amazon listings.
Reality
New disallow rules for ChatGPT-User and OAI-SearchBot went into Amazon's robots.txt without an announcement. Amazon earns roughly $56 billion a year from advertising that bills only while shoppers are on its own pages.
Reality
Google-Extended and Applebot-Extended are robots.txt tokens that never issue a request, so the nine agents OpenAI, Anthropic and Perplexity document are the only AI traffic a WordPress site can confirm for itself.
Reality
Before starting a column about getting cited by LLMs, a developer curled medium.com/robots.txt and found GPTBot, ClaudeBot and six other agents disallowed. Common Crawl and Google-Extended are still allowed.
Reality
Cloudflare's Disallow AI Training control lets Googlebot and Applebot keep indexing a site while a directive Cloudflare writes into robots.txt tells them not to train on it. Bingbot is expected to honor it in early 2027.
Reality
Eight hand-kept lists sat behind every new page on pub-trivia.app. One typed array now feeds the sitemap, breadcrumbs and footer, with unit tests that fail the commit when a description runs long.
Reality
The AI Act's general-purpose AI obligations became enforceable in August, and the parts of them that touch a data pipeline depend on records only the collector can write while it is fetching. Penalties scale with turnover.
Reality
With no completed sales, nine live asks running from £35 to £240 and no route to the £26.99 shop price, a self-hosted pricing tool proposed £35.39 and recorded the missing check only in a log line.
Reality
A dev.to post sets out two ways assistant retrieval fails while ordinary search keeps working, one in robots.txt rules written per user-agent and one in copy that appears only after JavaScript runs. It offers no measurement.
Reality
The file exists so an agent can skip your documentation and read the install steps instead. Pandex checked 8,565 of them, found 237 references pointing at names anyone could register, and registered one.
Reality
One developer's cache-header diff read perfectly and still shipped a dead year-long cache, which is the argument for granting agents autonomy only where git revert plus a redeploy is the whole undo path.
Reality
A vendor that generates hreflang tags explains why one missing return link discards an entire cluster, silently, while every page in that cluster still passes inspection on its own.
Reality
Bot Preference Sync writes Cloudflare's AI bot settings into your robots.txt from the Free tier up. The file becomes a rendering of dashboard state, and Cloudflare's bot database decides what it says.
Reality
The managed robots.txt file lists eight crawlers. Three of them are the training half of a pair whose search half is left allowed, and Perplexity is not in the file at all.
Reality
Build · August 17, 2026 · 1 publisher
Build · August 15, 2026 · 1 publisher