Quality, provenance, and observability

How to keep a crawl corpus current, attributable, non-duplicative, safe, and measurable.

Return to the complete atlas, or read this six-article section in any order.

• Content freshnessContent freshness describes when meaningful facts were last reviewed or changed.
• Source provenanceProvenance records where a claim came from, when it was observed, and which transformations produced the presented form.
• Duplicate contentDuplicate content exposes the same or nearly the same material at more than one URL.
• Crawl trapsA crawl trap generates an unbounded or extremely large set of low-value URLs.
• Measurement qualityMeasurement quality is the degree to which telemetry supports the claim being made from it.
• Crawler-facing error budgetsAn error budget defines how much failure and latency a crawl surface can tolerate before measurements or discovery become unreliable.