robots policy cohort

Every target is deliberately linked, while robots.txt asks one or more crawler groups not to fetch it.

The pages contain no confidential information; robots.txt is being measured as a voluntary crawl policy, never treated as access control.

• GPTBot-specific exclusionPublic measurement target with a declared crawl exclusion.
• ClaudeBot-specific exclusionPublic measurement target with a declared crawl exclusion.
• PerplexityBot-specific exclusionPublic measurement target with a declared crawl exclusion.
• Generic exclusion APublic measurement target with a declared crawl exclusion.
• Generic exclusion BPublic measurement target with a declared crawl exclusion.
• Generic exclusion CPublic measurement target with a declared crawl exclusion.

Measurement marker: ARN-POLICY-HUB