other
OpenAI's training crawler, cited as the example of a bot that can be blocked in robots.txt without covering live-browsing retrieval.
In a 30,180-request test across roughly 1,000 pages, Googlebot followed JavaScript-injected links. GPTBot, ClaudeBot and PerplexityBot did not follow them at all.
Reality
The managed robots.txt file lists eight crawlers. Three of them are the training half of a pair whose search half is left allowed, and Perplexity is not in the file at all.
Reality
A one-afternoon test splits AI visibility in two: name recall lags funding and press by years, while category retrieval is already working for products the model cannot describe.
Reality
A developer read the current Terms of Service for every feed in his news digest. The clauses that would break a summarization pipeline clustered in VC-data sites and well-lawyered corporate blogs.
Reality
A developer's 34-day Caddy log audit found 40.3 assistant fetches a day against Googlebot's 35.8. None of that traffic appears in Search Console.
Reality
A dev.to explainer redraws the boundary most comparisons blur: access policy, crawl map, reading brief. Conflating them is how teams block themselves or expose what they meant to hide.
Reality