We found no statement about automated access in what we read.
| Agent | Rules it follows (RFC 9309) | Over the whole file |
|---|---|---|
| * | the * group | 650 Disallow rules; / allowed |
| Bingbot | its own group | 651 Disallow rules; / allowed |
| GPTBot | its own group | disallowed at / (line 37) |
| OAI-SearchBot | its own group | 652 Disallow rules; / allowed |
| ChatGPT-User | its own group | 1304 Disallow rules; / allowed |
| ClaudeBot | its own group | disallowed at / (line 37) |
| Google-Extended | its own group | disallowed at / (line 735), with 9 Allow rules |
| Applebot-Extended | its own group | disallowed at / (line 37) |
| PerplexityBot | its own group | 654 Disallow rules; / allowed |
| CCBot | its own group | disallowed at / (line 37) |
| Bytespider | its own group | disallowed at / (line 37) |
| meta-externalagent | its own group | disallowed at / (line 37) |
Not found by us: no API, feed or export.
Not found by us: terms of use.