HTTP behavior and efficiency

How status codes, redirects, validators, caching, compression, and negotiation shape a crawl.

Return to the complete atlas, or read this six-article section in any order.

• HTTP status semanticsHTTP status codes tell a crawler whether a request succeeded, moved, failed temporarily, or failed permanently.
• Redirect chainsRedirects tell clients that a resource is available at another URL.
• Conditional requestsConditional requests let a crawler ask whether a previously fetched representation has changed.
• Cache validatorsETag and Last-Modified values identify the version or modification time of a representation.
• Response compressionCompression reduces transferred bytes when the client advertises a supported content encoding.
• Content negotiationContent negotiation selects a representation using request headers such as Accept, Accept-Language, and Accept-Encoding.