Topic
Patterns of AI web crawlers like GPTBot, ClaudeBot and PerplexityBot: what they fetch, whether they render JavaScript, and how sites manage their access.
Cloudflare's Pay Per Use beta charges AI companies for each use of a publisher's page they report, one three-field JSON line per event. Publishers can license without negotiating, though the buyer defines the use, sets the price and keeps the count.
Reality
Cloudflare says AI-agent requests on its network grew more than 1,700% in a year as non-human traffic passed half the total. It now wants sites to identify and price agents as customers, after advising last year that new domains block AI training crawlers.
Reality
FoxyInvoice prerenders every public Angular route to static HTML at build time so crawlers that skip JavaScript get the full page text. A build step and a one-line drift check take the place of the always-on SSR server its team declined to run.
Reality
Google-Extended and Applebot-Extended are robots.txt tokens that never issue a request, so the nine agents OpenAI, Anthropic and Perplexity document are the only AI traffic a WordPress site can confirm for itself.
Reality
One operator's 30-day log counts 6,027 requests from Meta's training collector, 155 from the crawler documented for Meta AI search quality, and zero from the fetcher that fires when a person asks a question.
Reality
Suman Debnath ran a week-long answer-engine pass, watched ChatGPT name him two days later, and then spent most of the write-up arguing that the sequence is not a cause. The cross-device check and the three silent assistants are why that argument holds.
Reality
In a 30,180-request test across roughly 1,000 pages, Googlebot followed JavaScript-injected links. GPTBot, ClaudeBot and PerplexityBot did not follow them at all.
Reality