Crawl Field Notes — how machines read the web

Every page you publish is read twice: once by people, and once by machines. This is a small field guide to the second reader — the crawlers and AI agents that discover, fetch, and try to make sense of the open web.

Each note below takes one mechanism and shows it working (or failing) on this very site. Follow the links the way a crawler would, and you will walk the whole guide.

Start here

• How crawlers find pagesLinks, sitemaps, and the crawl frontier.
• The links your crawler can't seeWhy JavaScript-only navigation gets left behind.
• Structured dataGiving machines an explicit, unambiguous reading.
• robots.txt and the meta robots tagHow sites ask crawlers to behave.
• Sitemaps and orphan pagesThe pages nothing links to.
• Crawl depth and reading orderWhat deep pages risk losing.
• The same facts, five waysWhich content formats a crawler will fetch.
• Crawl budgetThe finite patience every crawler arrives with.
• To render, or not to renderWho pays to run your JavaScript.
• Crawler Knowledge AtlasThirty-six practical reference articles, each in HTML, Markdown, and JSON.
• The field notes archiveHow deep a crawler will paginate.
• A page behind a redirectWhich crawlers follow the 302.