Crawl Field Notes

Sitemap discovery

An XML sitemap is a canonical inventory of URLs a site would like crawlers to know about.

Atlas version atlas-2026-07-28-v1 · Section: Discovery and crawl frontiers

How it works

Crawlers may fetch the sitemap from robots.txt, a remembered location, or a webmaster tool. The sitemap is a hint: inclusion does not guarantee fetching or indexing.

How to validate it

Use absolute canonical URLs, accurate last-modified dates, and a cohort that has no other discovery route; observe sitemap and page requests as separate events.

Common failure

Listing duplicates, redirects, blocked URLs, and non-canonical variants creates ambiguous signals and spends crawl attention on low-value addresses.

Machine-readable editions

The same article is available as Markdown and JSON. These alternates are linked here but are not submitted through the sitemap or IndexNow.

Related topics