AI Web Crawlers

Automated clients that fetch public web pages for model training, retrieval and assistant browsing, usually announcing themselves through a user-agent string such as GPTBot or ClaudeBot.