Discovery and crawl frontiers

How crawlers learn that URLs exist, prioritize them, and avoid revisiting the same resource.

Return to the complete atlas, or read this six-article section in any order.

• Seed URLsSeed URLs are the starting addresses placed into a crawler before link extraction begins.
• The crawl frontierThe crawl frontier is the ordered queue of discovered URLs that have not yet been fetched.
• Internal link discoveryOrdinary HTML links are the most portable discovery mechanism because they are visible without executing JavaScript.
• Orphan pagesAn orphan page has no crawlable inbound link from the connected site graph.
• Sitemap discoveryAn XML sitemap is a canonical inventory of URLs a site would like crawlers to know about.
• URL normalizationURL normalization maps syntactic variants to a consistent identity before deduplication and scheduling.