The crawl frontier
The crawl frontier is the ordered queue of discovered URLs that have not yet been fetched.
How it works
Schedulers assign priority using signals such as host load, estimated value, freshness, link distance, prior failures, and crawl policy. Different ordering rules can produce different coverage under the same budget.
How to validate it
Persist discovered, queued, fetched, and parsed timestamps so queue delay and terminal skip reasons can be measured rather than inferred.
Common failure
A final list of fetched pages cannot explain whether a missing URL was never discovered, deprioritized, blocked, or exhausted by budget.
Machine-readable editions
The same article is available as Markdown and JSON. These alternates are linked here but are not submitted through the sitemap or IndexNow.