robots.txt:
AI / LLM Crawlers and Agents: Use of any robot, spider, service search/retrieval application, or other automated device, process or means to access, retrieve, copy, scrape, or index any portion of the service or any content is prohibited, except as expressly permitted by Yelp.
If you would like to inquire about crawling Yelp, please contact us at https://www.yelp.com/contact
If you would like to request permission to crawl or use Yelp content, contact us here: https://www.yelp.com/contact
| Agent | Rules it follows (RFC 9309) | Over the whole file |
|---|---|---|
| * | the * group | disallowed at / (line 364) |
| Googlebot | its own group | 227 Disallow rules; / allowed; not bound by 1 of the * Disallow rules |
| Bingbot | its own group | 128 Disallow rules; / allowed; not bound by 1 of the * Disallow rules |
| GPTBot | its own group | disallowed at / (line 361) |
| OAI-SearchBot | its own group | disallowed at / (line 340), with 3 Allow rules |
| ChatGPT-User | its own group | disallowed at / (line 350), with 2 Allow rules |
| ClaudeBot | its own group | disallowed at / (line 361) |
| Claude-SearchBot | its own group | disallowed at / (line 350), with 2 Allow rules |
| Claude-User | its own group | disallowed at / (line 350), with 2 Allow rules |
| Google-Extended | its own group | disallowed at / (line 361) |
| PerplexityBot | its own group | disallowed at / (line 350), with 2 Allow rules |
| CCBot | its own group | disallowed at / (line 361) |
| meta-externalagent | its own group | disallowed at / (line 361) |
Not found by us: no API, feed or export.