Crawl Field Notes

The same facts, marker in the meta description

Same dataset. This edition hides its format marker only in the page's meta description, the <meta name="description"> tag in the head, never in the visible body.

GPTBot

Operated by OpenAI. Primary purpose: Training & grounding. Honours robots.txt.

ClaudeBot

Operated by Anthropic. Primary purpose: Model training. Honours robots.txt.

PerplexityBot

Operated by Perplexity. Primary purpose: Live cited answers. Honours robots.txt.