crawlsignals · How we read · About, corrections and removal · Privacy · records.json

Instagram www.instagram.com · social

What it says about automated access

robots.txt:

Notice: Collection of data on Instagram through automated means is prohibited unless you have express written permission from Instagram and may only be conducted for the limited purpose contained in said permission.
automated-access · source, read 2026-10-03
All authorized user-agents listed on this page must comply with Meta’s Automated Data Collection Terms available at: https://www.facebook.com/legal/automated_data_collection_terms
automated-access, permission · source, read 2026-10-03

robots.txt https://www.instagram.com/robots.txt read 2026-10-03

Agent	Rules it follows (RFC 9309)	Over the whole file
*	the `*` group	disallowed at `/` (line 288)
Googlebot	its own group	11 Disallow rules; `/` allowed; not bound by 1 of the `*` Disallow rules
Bingbot	its own group	11 Disallow rules; `/` allowed; not bound by 1 of the `*` Disallow rules
GPTBot	its own group	disallowed at `/` (line 24)
ClaudeBot	its own group	disallowed at `/` (line 18)
Google-Extended	its own group	disallowed at `/` (line 21)
Applebot-Extended	its own group	disallowed at `/` (line 12)
PerplexityBot	its own group	disallowed at `/` (line 27)

With no group of their own, these follow the * group: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, CCBot, Bytespider, meta-externalagent.

Under RFC 9309 an agent with its own group follows only that group, so the * rules do not apply to it.

Official way in

Not found by us: no API, feed or export.

Documents

Automated Data Collection Terms terms · named in robots.txt, line 7 of www.facebook.com · not read by us: the robots.txt of www.facebook.com prohibits crawling in its comments, so our crawler reads nothing else there

Nothing here is legal advice. A mistake? Write to hello@crawlsignals.com (corrections).