X-Robots-Tag
X-Robots-Tag carries indexing directives in an HTTP response header.
How it works
It can govern non-HTML resources such as PDFs and can target specific crawler names, while using directive semantics similar to meta robots.
How to validate it
Capture response headers before redirects and after the final response; test resource types that cannot contain an HTML head.
Common failure
A proxy or CDN configuration can accidentally apply noindex to an entire path when the intent covered only one file type.
Machine-readable editions
The same article is available as Markdown and JSON. These alternates are linked here but are not submitted through the sitemap or IndexNow.