# The crawl frontier

Canonical HTML: /atlas/crawl-frontier
Atlas version: atlas-2026-07-28-v1
Category: Discovery and crawl frontiers

## Definition

The crawl frontier is the ordered queue of discovered URLs that have not yet been fetched.

## How it works

Schedulers assign priority using signals such as host load, estimated value, freshness, link distance, prior failures, and crawl policy. Different ordering rules can produce different coverage under the same budget.

## How to validate it

Persist discovered, queued, fetched, and parsed timestamps so queue delay and terminal skip reasons can be measured rather than inferred.

## Common failure

A final list of fetched pages cannot explain whether a missing URL was never discovered, deprioritized, blocked, or exhausted by budget.

## Related topics

- [Seed URLs](/atlas/seed-urls)
- [crawl-budget](/atlas/crawl-budget)