Crawl Field Notes

Content negotiation

Content negotiation selects a representation using request headers such as Accept, Accept-Language, and Accept-Encoding.

Atlas version atlas-2026-07-28-v1 · Section: HTTP behavior and efficiency

How it works

One URL can return different media types or languages, but caches and crawlers need explicit Vary metadata and stable canonical relationships.

How to validate it

Replay controlled header matrices and hash every returned representation so variation is observable rather than guessed from the URL.

Common failure

Serving materially different facts to crawler user-agents creates cloaking risk and destroys reproducibility.

Machine-readable editions

The same article is available as Markdown and JSON. These alternates are linked here but are not submitted through the sitemap or IndexNow.

Related topics