Crawler and indexing directives

The standards and hints sites use to request crawling, indexing, following, and canonicalization behavior.

Return to the complete atlas, or read this six-article section in any order.

• robots.txtrobots.txt publishes path-based crawl rules for identified user-agent groups at the origin root.
• Meta robotsA meta robots element gives indexing and following instructions inside an HTML page that has already been fetched.
• X-Robots-TagX-Robots-Tag carries indexing directives in an HTTP response header.
• Canonical linksA canonical link identifies the preferred representative among duplicate or near-duplicate URLs.
• nofollow on linksThe nofollow relationship marks a link whose destination should not receive the normal endorsement signal.
• hreflang alternateshreflang annotations connect language or regional variants of substantially equivalent content.