robots.txt
robots.txt publishes path-based crawl rules for identified user-agent groups at the origin root.
How it works
A compliant crawler retrieves the file, chooses the most specific matching group, and evaluates allow and disallow patterns before requesting a URL.
How to validate it
Observe the robots request, effective user-agent claim, selected rule, and blocked URL as distinct records; use authentication for actual confidentiality.
Common failure
robots.txt is not access control, does not guarantee de-indexing, and may reveal the names of paths a site hoped to keep obscure.
Machine-readable editions
The same article is available as Markdown and JSON. These alternates are linked here but are not submitted through the sitemap or IndexNow.