# Crawl traps

Canonical HTML: /atlas/crawl-traps
Atlas version: atlas-2026-07-28-v1
Category: Quality, provenance, and observability

## Definition

A crawl trap generates an unbounded or extremely large set of low-value URLs.

## How it works

Calendars, faceted filters, session parameters, recursive paths, and endlessly paginated endpoints can keep producing novel strings for the frontier.

## How to validate it

Track URL-pattern growth, unique-content hashes, depth, and marginal new information; stop test crawlers with explicit budgets.

## Common failure

Blocking a trap only after millions of variants have been published leaves known URLs in external crawl queues for a long time.

## Related topics

- [Pagination](/atlas/pagination)
- [Duplicate content](/atlas/duplicate-content)