other
GPTBot is OpenAI's web crawler that fetches pages to gather training data for its AI models; site owners can block it via robots.txt.
No evidence-backed relationships are recorded.
OAI-SearchBot and GPTBot began rendering JavaScript on September 25, 2026, according to IP-verified logs from a Japanese Next.js marketplace. OpenAI has not announced it, so the case rests on prefetch URLs that only a JavaScript run can produce.
Reality
Googlebot got 0 characters inside a client-rendered blog's root div, its developer found after Google crawled 270 of the posts and indexed none. A bot-only branch in an existing Netlify edge function now hands crawlers the rendered article and JSON-LD.
Reality
FoxyInvoice prerenders every public Angular route to static HTML at build time so crawlers that skip JavaScript get the full page text. A build step and a one-line drift check take the place of the always-on SSR server its team declined to run.
Reality
Amazon's robots.txt now disallows ChatGPT-User and OAI-SearchBot, and in ZDNET's tests the shopping agent sent a buyer to Walmart, eBay and a retailer called Soaplicity instead of Amazon listings.
Reality
New disallow rules for ChatGPT-User and OAI-SearchBot went into Amazon's robots.txt without an announcement. Amazon earns roughly $56 billion a year from advertising that bills only while shoppers are on its own pages.
Reality
DataDome says malicious automated traffic grew 124% in the year to June. Its June test of 21,491 popular sites found that claiming a trusted crawler's name is usually enough to get through the door.
Reality
A site logged twenty requests for its unreachable /.env file from a client naming itself ClaudeBot. GreyNoise counted 824 addresses scanning under forged AI crawler names, and saw no request for robots.txt in the whole campaign.
Reality
Google-Extended and Applebot-Extended are robots.txt tokens that never issue a request, so the nine agents OpenAI, Anthropic and Perplexity document are the only AI traffic a WordPress site can confirm for itself.
Reality
The llms.txt spec picked up link relations and subpath scope in a quiet 10 August revision. The fetch logs suggest the file gets read only when a client has already been told to look for it.
Reality
Profound raised $96 million at a $1 billion valuation in February and Adobe closed on Semrush in April. Both deals assume presence in AI answers can be counted, and when ChatGPT names a brand without a link there is no referrer to count it.
Reality
Before starting a column about getting cited by LLMs, a developer curled medium.com/robots.txt and found GPTBot, ClaudeBot and six other agents disallowed. Common Crawl and Google-Extended are still allowed.
Reality
A dev.to post sets out two ways assistant retrieval fails while ordinary search keeps working, one in robots.txt rules written per user-agent and one in copy that appears only after JavaScript runs. It offers no measurement.
Reality
Debugging an agent endpoint has to start on the wire, because a client that reports a parse error is describing the last thing that touched the bytes rather than the machine that swapped them out.
Reality
A CDN rule aimed at AI crawlers matched the Vendor/Language User-Agent shape that the official OpenAI and Anthropic SDKs send by default. The harness meant to catch it sent an allowlisted string of its own and reported green.
Reality
A 41-day test switched off sitemaps and breadcrumbs so client-side navigation was the only way in, and the crawlers behind ChatGPT and Claude never arrived. Googlebot did, then needed 41 days to notice the fix.
Reality
Promptwatch ties the collapse to an August 8 jump in domain-scoped fan-out queries. The data is provisional. The dependency it exposes is not.
Reality
In a 30,180-request test across roughly 1,000 pages, Googlebot followed JavaScript-injected links. GPTBot, ClaudeBot and PerplexityBot did not follow them at all.
Reality
The managed robots.txt file lists eight crawlers. Three of them are the training half of a pair whose search half is left allowed, and Perplexity is not in the file at all.
Reality
A one-afternoon test splits AI visibility in two: name recall lags funding and press by years, while category retrieval is already working for products the model cannot describe.
Reality
A developer read the current Terms of Service for every feed in his news digest. The clauses that would break a summarization pipeline clustered in VC-data sites and well-lawyered corporate blogs.
Reality
Build · August 16, 2026 · 1 publisher
Build · August 15, 2026 · 1 publisher