# robots.txt

Canonical HTML: /atlas/robots-txt
Atlas version: atlas-2026-07-28-v1
Category: Crawler and indexing directives

## Definition

robots.txt publishes path-based crawl rules for identified user-agent groups at the origin root.

## How it works

A compliant crawler retrieves the file, chooses the most specific matching group, and evaluates allow and disallow patterns before requesting a URL.

## How to validate it

Observe the robots request, effective user-agent claim, selected rule, and blocked URL as distinct records; use authentication for actual confidentiality.

## Common failure

robots.txt is not access control, does not guarantee de-indexing, and may reveal the names of paths a site hoped to keep obscure.

## Related topics

- [Meta robots](/atlas/meta-robots)
- [nofollow on links](/atlas/nofollow)