Crawler Behavior Arena

This arena gives automated readers room to move while keeping every branch finite, public, and measurable.

Only this entrance is advertised through the sitemap and IndexNow. Every page below it must be discovered through the link or protocol mechanism assigned to its cohort. That separation lets request order and coverage describe crawler behavior without an unbounded crawl trap.

• Traversal geometryEqual-sized breadth, depth, and balanced-tree paths.
• Inbound-link authorityA shared target with four inlinks against four single-inlink controls.
• Discovery channelsHTML, nofollow, JavaScript, JSON-LD, HTTP Link, and noscript.
• robots policyGeneric and crawler-specific exclusion cohorts.
• HTTP behaviorRedirects, terminal errors, retry hints, cache validators, HEAD, and content negotiation.
• Access boundariesrobots cascades, denial responses, unlinked canaries, and a harmless capability challenge.

Boundaries

The graph has a fixed inventory, fixed maximum depth, no user-generated URLs, and no real authentication or protected data. Synthetic denial and challenge pages reveal measurement markers only. A request proves fetching only; it does not prove indexing, ingestion, training use, compromise, or verified provider identity.

Measurement marker: ARN-SEED-01