# robots.txt โ€” sewercleaningcompany.com # One group per user-agent. Zero duplicates. Zero conflicts. # RFC 9309 requires crawlers to merge duplicate groups; not all resolve # allow/disallow ties the same way, so we never create a tie. # # ๐Ÿšฉ THIS FILE IS ONLY HALF THE STORY. Cloudflare PREPENDS a managed block # ahead of it that Disallows GPTBot, ClaudeBot, CCBot, Google-Extended, # Bytespider, meta-externalagent, Amazonbot and Applebot-Extended, and sets # Content-Signal: ai-train=no. The origin file below cannot reliably override # it. H2 in HUMAN-ACTIONS.md turns that block off โ€” until that lands, verify # what is actually served: # curl -s https://sewercleaningcompany.com/robots.txt | head -40 # If you see "# BEGIN Cloudflare Managed content", H2 is still open. # # ๐Ÿšฉ A 5xx on this file means COMPLETE DISALLOW (RFC 9309 ยง2.3.1.4), and # crawlers cache that failure. Needs <=5-minute synthetic monitoring. # --------------------------------------------------------------- # Default posture # --------------------------------------------------------------- User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes, use=full Allow: / # --------------------------------------------------------------- # Search engines # --------------------------------------------------------------- User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Video Allow: / User-agent: Googlebot-News Allow: / # Google-Extended controls Gemini app grounding + Gemini model training ONLY. # Per Google's own docs it does NOT affect Search, AI Overviews, AI Mode or # ranking. Blocking it costs Gemini visibility and buys nothing here. User-agent: Google-Extended Allow: / User-agent: bingbot Allow: / # Named explicitly because Applebot FALLS BACK TO GOOGLEBOT'S RULES if it is # not mentioned. Same for Amzn-SearchBot below. Naming removes that indirection. User-agent: Applebot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Baiduspider Allow: / User-agent: YandexBot Allow: / User-agent: Slurp Allow: / User-agent: Bravebot Allow: / # --------------------------------------------------------------- # AI answer engines โ€” live retrieval. # Blocking any of these makes the site invisible in that assistant. # --------------------------------------------------------------- User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Meta: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and # link to your content in Meta AI's responses." User-agent: meta-webindexer Allow: / User-agent: meta-externalagent Allow: / User-agent: meta-externalfetcher Allow: / User-agent: Amzn-SearchBot Allow: / User-agent: Amzn-User Allow: / User-agent: DuckAssistBot Allow: / User-agent: MistralAI-User Allow: / User-agent: cohere-ai Allow: / User-agent: YouBot Allow: / User-agent: PetalBot Allow: / User-agent: kagi-fetcher Allow: / # --------------------------------------------------------------- # Retrieval infrastructure. No consumer assistants of their own, # but they supply retrieval to agents built by others. # --------------------------------------------------------------- User-agent: ExaBot Allow: / User-agent: TavilyBot Allow: / User-agent: FirecrawlAgent Allow: / User-agent: Cloudflare-AutoRAG Allow: / # --------------------------------------------------------------- # Training crawlers. Allowed deliberately: blocking costs zero # visibility, and being in training corpora builds the entity # association that later surfaces in parametric answers. # --------------------------------------------------------------- User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: CCBot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Amazonbot Allow: / User-agent: AI2Bot Allow: / # --------------------------------------------------------------- # Social preview fetchers. Do NOT block โ€” you lose rich previews # across Facebook, Instagram, Messenger, WhatsApp and Apple Messages. # --------------------------------------------------------------- User-agent: facebookexternalhit Allow: / User-agent: Twitterbot Allow: / User-agent: LinkedInBot Allow: / User-agent: Slackbot-LinkExpanding Allow: / User-agent: WhatsApp Allow: / # --------------------------------------------------------------- # Sitemaps # --------------------------------------------------------------- Sitemap: https://sewercleaningcompany.com/sitemap.xml