Crawler access controls

The mechanisms site owners use to declare or enforce which automated clients may fetch their pages, including robots.txt directives, per-vendor opt-out tokens, IP range verification and edge blocking.