Google robots.txt token scoped to training and grounding future Gemini models; has no separate user agent string and does not affect Search inclusion or ranking.
Cloudflare's September 15 split of AI bot controls into Search, Agent and Training can block Googlebot when a site blocks Training. Site owners who depend on organic search should check their settings, a dev.to analysis says.
Reality
Google-Extended and Applebot-Extended are robots.txt tokens that never issue a request, so the nine agents OpenAI, Anthropic and Perplexity document are the only AI traffic a WordPress site can confirm for itself.
Reality
Before starting a column about getting cited by LLMs, a developer curled medium.com/robots.txt and found GPTBot, ClaudeBot and six other agents disallowed. Common Crawl and Google-Extended are still allowed.
Reality
Aggarwal et al. report visibility gains of up to roughly 40% inside generated answers, but the benchmark supplies the source documents, so the number scores selection and leaves crawl and index eligibility upstream of anything you can format.
Reality
The managed robots.txt file lists eight crawlers. Three of them are the training half of a pair whose search half is left allowed, and Perplexity is not in the file at all.
Reality