# X-Robots-Tag

Canonical HTML: /atlas/x-robots-tag
Atlas version: atlas-2026-07-28-v1
Category: Crawler and indexing directives

## Definition

X-Robots-Tag carries indexing directives in an HTTP response header.

## How it works

It can govern non-HTML resources such as PDFs and can target specific crawler names, while using directive semantics similar to meta robots.

## How to validate it

Capture response headers before redirects and after the final response; test resource types that cannot contain an HTML head.

## Common failure

A proxy or CDN configuration can accidentally apply noindex to an entire path when the intent covered only one file type.

## Related topics

- [Meta robots](/atlas/meta-robots)
- [HTTP status semantics](/atlas/http-status)