# robots.txt for https://totalshiftleft.ai # Updated: 2026-10-07 # Default rules for all crawlers # /_next/static/ (CSS, JS) and /_next/image (every optimised ) are allowed # explicitly: search engines render pages and index images from those paths, and # the longer Allow beats the shorter Disallow: /_next/ that follows. /api/og is the # 1200x630 social and Article image route, so it is allowed out of /api/ the same way. User-agent: * Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Allow: /api/og Disallow: /api/ Disallow: /thank-you Disallow: /*?q=* Disallow: /*?utm_* Disallow: /*?gclid=* Disallow: /*?fbclid=* # Google User-agent: Googlebot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Allow: /api/og Disallow: /api/ Disallow: /thank-you User-agent: Googlebot-Image Allow: /images/ Allow: /brand/ Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ User-agent: Googlebot-News Allow: /blog/ # Microsoft / Bing User-agent: Bingbot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Allow: /api/og Disallow: /api/ Disallow: /thank-you # DuckDuckGo User-agent: DuckDuckBot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Allow: /api/og Disallow: /api/ # Yandex User-agent: YandexBot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Allow: /api/og Disallow: /api/ # --------------------------------------------------------------------------- # AI crawlers (answer engines / training). We allow these to increase # visibility in AI-powered search. Block sensitive paths only. # --------------------------------------------------------------------------- # OpenAI – ChatGPT web browsing (user-triggered) User-agent: ChatGPT-User Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # OpenAI – GPT model training crawler User-agent: GPTBot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ Crawl-delay: 5 # OpenAI – OAI search index User-agent: OAI-SearchBot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # Anthropic – Claude web fetcher (user-triggered) User-agent: Claude-User Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # Anthropic – Claude search index User-agent: Claude-SearchBot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # Anthropic – training crawler User-agent: ClaudeBot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ Crawl-delay: 5 User-agent: anthropic-ai Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # Perplexity – answer engine User-agent: PerplexityBot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ User-agent: Perplexity-User Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # Google – extended/AI training (Gemini, Vertex) User-agent: Google-Extended Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # Apple – extended/AI training User-agent: Applebot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ User-agent: Applebot-Extended Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # Common Crawl User-agent: CCBot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ Crawl-delay: 10 # Meta / Facebook AI User-agent: FacebookBot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ User-agent: Meta-ExternalAgent Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # ByteDance User-agent: Bytespider Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # Amazon User-agent: Amazonbot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # Mistral User-agent: MistralAI-User Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # Cohere User-agent: cohere-ai Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # --------------------------------------------------------------------------- # SEO tools — allowed so our own audits and rank tracking can crawl the site. # (Re-block any of these if you are not actively using them.) # --------------------------------------------------------------------------- User-agent: SemrushBot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ User-agent: AhrefsBot Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/ Disallow: /api/ # --------------------------------------------------------------------------- # Block notorious scrapers / non-reciprocal bots # --------------------------------------------------------------------------- User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: PetalBot Disallow: / # --------------------------------------------------------------------------- # Sitemaps & LLM metadata # --------------------------------------------------------------------------- Sitemap: https://totalshiftleft.ai/sitemap.xml # LLM discovery endpoints # https://totalshiftleft.ai/llms.txt (summary) # https://totalshiftleft.ai/llms-full.txt (comprehensive) # --------------------------------------------------------------------------- # Content reuse / citation policy # --------------------------------------------------------------------------- # Content on this site may be cited, summarized, and referenced by AI systems # (ChatGPT, Claude, Perplexity, Gemini, Google AI Overviews, etc.) with clear # attribution to "Total Shift Left" and a working link back to the source URL # on totalshiftleft.ai. Verbatim large-scale reproduction without attribution # is not permitted. # # Security disclosures: https://totalshiftleft.ai/.well-known/security.txt # Sitemap: https://totalshiftleft.ai/sitemap.xml # RSS feed: https://totalshiftleft.ai/rss.xml # JSON feed: https://totalshiftleft.ai/feed.json