# ReelForge AI - robots.txt # https://reelforgeai.io # # 2026 policy: allow classical search bots AND the AI bots that influence # answer-engine citations. The 2024-era split of "search bots = good, # training bots = bad" no longer matches reality — Anthropic (ClaudeBot) # and Google (Google-Extended) use a single bot family for both training # and live citation. Blocking them removes us from Claude.ai and Gemini # answers entirely, even though Search still works. # # Per Anthropic docs (support.claude.com/en/articles/8896518), Anthropic # operates three bots: ClaudeBot, Claude-User, Claude-SearchBot. # Per Google docs (developers.google.com/search/docs/crawling-indexing/ # google-common-crawlers), Google-Extended gates Gemini Apps + grounding. # Per OpenAI docs (platform.openai.com/docs/bots), GPTBot, OAI-SearchBot, # and ChatGPT-User are distinct surfaces. # # ByteDance bots (Bytespider, Doubaobot, TikTokSpider) reportedly ignore # robots.txt and offer no Western-market citation value — enforce at the # WAF layer for real protection. The entries below are advisory only. # --------------------------------------------------------------------------- # Default — allow, but block app/auth/admin surfaces # --------------------------------------------------------------------------- User-agent: * Allow: / Disallow: /api/ Disallow: /*/opengraph-image Disallow: /*/twitter-image Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /schedules Disallow: /calendar Disallow: /settings Disallow: /approvals Disallow: /social Disallow: /verify-email Disallow: /reset-password Disallow: /setup-password Disallow: /forgot-password Disallow: /welcome Disallow: /admin Crawl-delay: 1 # Explicit public paths Allow: /privacy Allow: /terms Allow: /contact Allow: /become-affiliate Allow: /affiliates Allow: /api-docs Allow: /pricing # --------------------------------------------------------------------------- # Classical search crawlers — full allowlist # --------------------------------------------------------------------------- User-agent: Googlebot Allow: / Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /settings User-agent: Bingbot Allow: / Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /settings User-agent: DuckDuckBot Allow: / Disallow: /api/ User-agent: Slurp Allow: / Disallow: /api/ User-agent: Yandex Allow: / Disallow: /api/ # --------------------------------------------------------------------------- # AI ANSWER-ENGINE crawlers — allow on public content # --------------------------------------------------------------------------- # These bots drive citations in ChatGPT, Claude, Gemini, Meta AI, # Perplexity, and the long tail of models that ingest Common Crawl. # Scoped to public marketing/content surfaces; auth/app paths blocked. # OpenAI — training, search index, and on-demand fetch User-agent: GPTBot Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /settings User-agent: OAI-SearchBot Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /settings User-agent: ChatGPT-User Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /settings # Anthropic — three bots per support.claude.com/en/articles/8896518. # ClaudeBot feeds the corpus Claude reasons over; Claude-User is the # on-demand fetch when a user pastes a URL into Claude.ai; Claude-SearchBot # is the index behind Claude's web-search citations. User-agent: ClaudeBot Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /settings User-agent: Claude-User Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ User-agent: Claude-SearchBot Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /settings # Perplexity — search index and on-demand fetch User-agent: PerplexityBot Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ User-agent: Perplexity-User Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ # Google — Google-Extended gates Gemini Apps and grounding. It does # NOT affect Search inclusion (Googlebot above handles that) and does # NOT gate AI Overviews. Blocking it removed us from Gemini answers. User-agent: Google-Extended Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /settings User-agent: GoogleOther Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /settings # Apple Intelligence / Siri search User-agent: Applebot-Extended Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos User-agent: Applebot Allow: / Disallow: /api/ # Meta AI / WhatsApp AI / Instagram AI — Meta-ExternalAgent is the # training+indexing bot; Meta-ExternalFetcher is the on-demand fetch # for agentic AI tasks. Meta docs note both may bypass robots.txt. User-agent: Meta-ExternalAgent Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /settings User-agent: Meta-ExternalFetcher Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ # Common Crawl — corpus ingested by DeepSeek, Mistral, Cohere, and # the long tail of LLMs that don't run their own crawlers. Blocking # this cuts us out of every model in that category. User-agent: CCBot Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ Disallow: /dashboard Disallow: /create Disallow: /videos Disallow: /settings # DuckDuckGo AI Assist User-agent: DuckAssistBot Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ # Amazon (Alexa, Amazon AI) User-agent: Amazonbot Allow: /blog Allow: /niche Allow: /compare Allow: /alternative Allow: /tools Allow: /platforms Allow: /use-cases Allow: /shadowban Allow: /shadowban-recovery Allow: /pricing Allow: /about Allow: /api-docs Allow: /llms.txt Allow: /llms-full.txt Disallow: /api/ # --------------------------------------------------------------------------- # Blocked — abusive scrapers and bots with no answer-engine value in # our market. ByteDance bots reportedly ignore robots.txt; for real # protection enforce at the WAF (Cloudflare AI Crawl Control etc.). # --------------------------------------------------------------------------- User-agent: Bytespider Disallow: / User-agent: bytespider Disallow: / User-agent: Doubaobot Disallow: / User-agent: TikTokSpider Disallow: / User-agent: FacebookBot Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Omgili Disallow: / User-agent: Timpibot Disallow: / # --------------------------------------------------------------------------- # Sitemap locations # --------------------------------------------------------------------------- Sitemap: https://reelforgeai.io/sitemap.xml Sitemap: https://reelforgeai.io/sitemap_index.xml Sitemap: https://reelforgeai.io/blog-sitemap.xml Sitemap: https://reelforgeai.io/niche-sitemap.xml Sitemap: https://reelforgeai.io/compare-sitemap.xml Sitemap: https://reelforgeai.io/pages-sitemap.xml