A page robots.txt asks crawlers to skip

The site's robots.txt disallows everything under /private/. If you reached this page as a crawler, you did not honour that rule.

It is deliberately linked from the guide so a crawler could find it — the only question is whether it obeys the request not to fetch it. Respecting robots.txt is voluntary; this page measures who actually does.