AI Crawlers and robots.txt

The automated fetchers that AI companies run to read the web, and the robots.txt conventions site owners use to allow or refuse them. Recurring questions include which agents identify themselves, what they render, and how access policy is enforced.