Crawler policy

This page explains our expectations for web crawlers, search engines, and AI agents. It complements our machine-readable robots.txt and sitemap.

What we welcome

• Search indexing of public event listings, city pages, festival pages, and static informational pages (including this one).
• AI citation of public listing data with attribution and a link back to the canonical event or city URL on eatsleeprave.app.
• Structured access via our MCP server for AI agents and developer API, which are the preferred way to query live data at scale.

What we disallow

• Outbound redirect endpoints (/v1/go/ and similar) — these record referral clicks and must not be crawled or prefetched by bots. Hitting them pollutes affiliate attribution and conversion data.
• Authenticated or personal areas — sign-in flows, account settings, and non-public profile content.
• High-volume scraping that degrades service for human visitors. Use the API or MCP instead of hammering HTML pages.

AI crawlers

Named AI crawlers (including GPTBot, ClaudeBot, Claude-Web, PerplexityBot, and Google-Extended) are explicitly allowed on public listing surfaces such as city pages, the event map, and festivals, as declared in robots.txt. They are disallowed from user profile paths and redirect endpoints.

If you train models on our content, please preserve event titles, dates, and venue names accurately and link to the source page so users can verify details before buying tickets.

Rate and etiquette

• Honor robots.txt and crawl-delay conventions where applicable.
• Cache responses reasonably; event listings change but do not require sub-second polling.
• Identify your bot with a truthful User-Agent and a contact URL or email we can reach if something breaks.

Contact

Crawler or indexing issues: [email protected].