Crawl budget and what it buys

No crawler reads everything. Each arrives with a rough budget — a sense of how many requests this site is worth right now — and spends it, page by page, until it moves on.

Budget is set by signals a site barely controls: how fast it responds, how often it changes, how much other sites link to it. It is then spent breadth-first from the entry points, which is why depth and discovery decide so much.

The practical failure is silent: a crawler burns its budget on low-value pages — filters, session URLs, endless pagination — and leaves before it reaches what mattered. No error; you simply never get read. See also how rendering multiplies the cost of every page.