Crawl Field Notes

The same facts, revealed only by JavaScript

Same dataset again, but this edition's format marker is not in the HTML source at all. A script writes it into the page after load, so only a crawler that actually executes JavaScript will ever see it.

Format marker: (set by JavaScript on render)

GPTBot

Operated by OpenAI. Primary purpose: Training & grounding. Honours robots.txt.

ClaudeBot

Operated by Anthropic. Primary purpose: Model training. Honours robots.txt.

PerplexityBot

Operated by Perplexity. Primary purpose: Live cited answers. Honours robots.txt.