Crawl traps
A crawl trap generates an unbounded or extremely large set of low-value URLs.
How it works
Calendars, faceted filters, session parameters, recursive paths, and endlessly paginated endpoints can keep producing novel strings for the frontier.
How to validate it
Track URL-pattern growth, unique-content hashes, depth, and marginal new information; stop test crawlers with explicit budgets.
Common failure
Blocking a trap only after millions of variants have been published leaves known URLs in external crawl queues for a long time.
Machine-readable editions
The same article is available as Markdown and JSON. These alternates are linked here but are not submitted through the sitemap or IndexNow.