crawlsignals · How we read · About, corrections and removal · Privacy · records.json
X x.com · social
What it says about automated access
robots.txt:
Wait 1 second between successive requests.
automated-access · source, read 2026-10-03
| Agent | Rules it follows (RFC 9309) | Over the whole file |
|---|
| * | the * group | disallowed at / (line 100) |
| Googlebot | its own group | 18 Disallow rules; / allowed; not bound by 2 of the * Disallow rules |
| Bingbot | its own group | 18 Disallow rules; / allowed; not bound by 2 of the * Disallow rules |
| Google-Extended | its own group | disallowed at / (line 75) |
| meta-externalagent | its own group | disallowed at / (line 89) |
With no group of their own, these follow the * group: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Applebot-Extended, PerplexityBot, CCBot, Bytespider.
Under RFC 9309 an agent with its own group follows only that group, so the * rules do not apply to it.
Official way in
Not found by us: no API, feed or export.
Documents
Not found by us: terms of use.
Nothing here is legal advice. A mistake? Write to hello@crawlsignals.com (corrections).