crawlsignals · How we read · About, corrections and removal · Privacy · records.json
Google www.google.com · reference
What it says about automated access
robots.txt:
Crawlers of certain social media sites are allowed to access page markup when google.com/imgres* links are shared. To learn more, please contact images-robots-allowlist@google.com.
automated-access, permission · source, read 2026-10-03
| Agent | Rules it follows (RFC 9309) | Over the whole file |
|---|
| * | the * group | 169 Disallow rules; / allowed |
With no group of their own, these follow the * group: Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent.
Official way in
Not found by us: no API, feed or export.
Documents
Not found by us: terms of use.
Nothing here is legal advice. A mistake? Write to hello@crawlsignals.com (corrections).