About

crawlsignals is for anyone who asks, before a project, what a website says about being read by machines. It is free and carries no advertising. No site pays to be listed, ranked or removed, and every site goes through the same steps.

• Which sites: a selection, not a sample of the web, so a count on these pages describes only the sites listed. They are sites people often ask about before scraping, building an AI product or reusing data (large platforms, publishers, reference, developer and public-data sites), and a smaller set of Norwegian sites from our first run. Anyone may send a site's address to add it.
• Contact: one address for everything, hello@crawlsignals.com. We aim to answer within 14 days.
• Adding a site takes its address, sent by anyone. We find its documents ourselves, as described under How we read. What we cannot find shows as "not found by us", and the site's owner may then send us the link. Sending an address again asks for a re-check.
• Corrections: if an entry misquotes a page, misses a newer version or shows text the site no longer has, write to us with the page's address. We judge a correction on the evidence, whether the text is on the page, not on who asks, so no proof of ownership is needed. We re-read the page the way it was first read, and the entry shows the new date. Where the mistake was ours, the entry says so, with the date. Which passages are quoted follows the same checklist for every site and does not change on request.
• Removing a site: an owner can ask us to remove their site. Write to us with the site's address, and we send a confirmation to an address on the site's own domain. Once it is confirmed, we remove the entry and our fetcher no longer reads the site. The index then shows "removed at the site's request" with the date.
• Who makes it: the makers of modelsentiment.com, which collects posts and comments from AI subreddits through Reddit's RSS feeds. Reddit's entry, its robots.txt included, is read the same way as every other site's.
• Data licence: our records (the fields we write, the readings we compute and the compilation) are under CC BY 4.0: reuse them, crediting crawlsignals.com. Text quoted from a site, its robots.txt included, belongs to that site's owner and is not under our licence. The records are one file, records.json. Our own robots.txt allows every crawler.

Privacy

• The controller is the operator of crawlsignals.com, in Norway, reachable at hello@crawlsignals.com.
• This page sets no cookies, runs no scripts and counts no visitors. Our server keeps no log of requests. Cloudflare (USA) carries traffic to it and processes your IP address to do so (its privacy policy).
• If you write to us, Cloudflare passes your message on to our mailbox at Microsoft (USA), and we keep it to answer and as the record of what was asked: for two years, except a message that confirmed a removal, which we keep while the removal stands. We never publish who wrote to us; a removal shows only that the site asked, and when.
• We do this on our legitimate interest in running the site and answering what is sent to us (GDPR Art. 6(1)(f)), and you may object to it.
• You can ask to see, correct or delete what we hold about you at hello@crawlsignals.com, and you can complain to Datatilsynet, the Norwegian data protection authority.