Where websites share their data, and on what terms.

How we read

• What we fetch: robots.txt and the addresses it names; the home page, read only for the links in its footer and never quoted; pages those links lead to whose address or link text names terms, legal, conditions, privacy, developers, a licence (or the Norwegian vilkår, betingelser, lisens), or whose link text names an API or data, and one more such link from those pages; /.well-known/ files and /llms.txt. Nothing else, unless a person approved that address: we never guess addresses no page links to. Never listings, posts, profiles, search results or data. A document in several languages is read once; a PDF is read as its text.
• How we fetch: one request at a time, at least two seconds apart per host (longer if its robots.txt asks), as crawlsignals/0.1 (+https://crawlsignals.com), with robots.txt obeyed for every page and every redirect. A page we are refused is never fetched another way.
• robots.txt is the file as our crawler got it on the date shown; a site may serve other crawlers a different one. It is described as RFC 9309 reads it, over the whole file: an agent with its own group follows only that group, not the * rules; the longest matching rule wins; a 4xx answer means no rules; a server error means every path is disallowed. Example paths are only examples.
• Our crawler is stricter than the standard. Only a 404 or 410 counts as "no robots.txt"; any other refusal (such as 403, 418 or 429) we treat as every path disallowed, because refusing a crawler its robots.txt is a signal too. Where robots.txt's comments prohibit crawling, we quote that and still read the documents listed above where its rules allow our crawler.
• Not read by us marks a page we did not read: a refusal, no answer, a page that needs JavaScript, a format we do not read. It is our gap, not a statement about the site, and nobody reads it another way: where a site's robots.txt or server refuses our crawler, the page says so and gives the reason.
• Quotes are copied from the page as read on the date shown, and a program checks that each appears word for word, on word boundaries, in the saved text of a successful answer.
• Open by design means an official API, feed or export that anyone, or anyone with a free key, may use, with the data under either a standard licence (CC0, CC BY, CC BY-SA, NLOD, ODbL, EU Commission Decision 2011/833) or the site's own terms granting reuse, each quoted. Meet these and send your site's address to hello@crawlsignals.com for a re-check.
• Nothing here is legal advice, and nothing says what you may do. We describe what each site says.