How crawlers find pages

A crawler cannot read a page it never finds. Discovery — not fetching — is the quiet bottleneck of the whole enterprise.

Most discovery is just link-following. A crawler starts from a set of known URLs, fetches each one, extracts the <a href> links it contains, and adds the new ones to a queue known as the crawl frontier. It works through that queue breadth-first or by priority, and the frontier grows and shrinks as pages are found and visited.

The second source is the sitemap: an explicit list a site hands over, naming pages it wants found — including ones nothing links to. Between links and sitemaps, a well-connected site is easy to map. A poorly linked one hides in plain sight.

Two failure modes are worth their own notes: links that only exist after JavaScript runs, which many crawlers never see, and pages buried so deep that the frontier reaches them late, or never.