Fix guide · info · robots_blocks_ai_crawlers
robots.txt blocks AI crawlers
Your robots.txt has explicit Disallow rules for GPTBot, ClaudeBot, PerplexityBot, or Google-Extended.
Why it matters
Your content won't appear in AI search results (ChatGPT, Claude, Perplexity, Google AI Overviews). For most sites in 2026, this is a missed distribution channel rather than a security issue.
How to fix it
If you want AI crawlers to index you, remove the Disallow rules for those user-agents. Recommended robots.txt:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: cohere-ai
Allow: /If you have legal reasons to block AI training (compliance, IP), document them in your terms of service — robots.txt is not legally enforceable.
Did vibecheck flag this on your app?
If you reached this page from a vibecheck inspection report, the redacted match in your scan output is the exact string we found in your bundle. After applying the fix above, run the inspection again — the finding should clear.
Run another inspection