ClaudeBot
ClaudeBot is Anthropic's web crawler that indexes content for Claude AI. Combined with anthropic-ai, it gathers content for Claude's knowledge.
Why It Matters for GEO
Blocking ClaudeBot prevents your content from being cited by Claude. Enable both ClaudeBot and anthropic-ai for full Claude visibility.
Every day that ClaudeBot cannot access your site, opportunities to appear in Claude's responses are missed. Because Claude's audience skews toward professionals and enterprise users, those missed citations often represent high-value prospects who use Claude for vendor research and decision support.
How to Configure
# robots.txt
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
Practical Example
A technology recruitment agency checks their robots.txt and finds a generic block: User-agent: * Disallow: / followed by an exception only for Googlebot. Every AI crawler including ClaudeBot is blocked. After adding explicit Allow rules for ClaudeBot and anthropic-ai, Anthropic's bots begin crawling their site within days. Two months later, when senior engineers ask Claude about specialized tech recruiters, the agency's content appears as a cited source — a channel that was completely closed before the simple robots.txt change.
Common Mistakes
- Using a wildcard block without exceptions:
User-agent: * Disallow: /is a common robots.txt pattern that blocks every bot not explicitly allowed. ClaudeBot must be listed as an explicit exception to get through. - Allowing ClaudeBot but forgetting anthropic-ai: Anthropic uses both user agents. Configuring only one may limit coverage. Always add both entries together.
- Not verifying after changes: After editing robots.txt, confirm the file is accessible at yourdomain.com/robots.txt and that the syntax is correct. A formatting error can silently block all bots.
- Assuming indexing is instant: After allowing ClaudeBot, it may take several weeks for Anthropic to recrawl your site and for the content to influence Claude's responses. Track your Claude citations monthly, not daily.