robots.txt and the meta robots tag

Two mechanisms, often confused, do opposite halves of the job. One governs fetching; the other governs what happens after a fetch.

robots.txt sits at the site root and asks well-behaved crawlers not to request certain paths at all. It is a request, not a wall — it stops polite crawlers from fetching, but it is not access control, and a URL disallowed here can still be listed if others link to it.

The meta robots tag (and the matching X-Robots-Tag header) works one level later: the crawler has already fetched the page, and the tag tells it whether to index the content and whether to follow the links. A page can be fetched, read, and then deliberately kept out of the index.

The AI-crawler era added new tokens to the same file — product-specific user-agents like GPTBot or ClaudeBot that a site can allow or disallow independently of search. The grammar is old; the negotiation it carries is very new.

This site disallows everything under /private/. There is a page in there, linked on purpose: whether a crawler fetches it anyway is a clean test of who honours the rules.