AI crawler user agents & robots.txt templates
The short version: AI companies run separate crawlers for search and for training, and robots.txt controls each one by name. Block the training crawlers if you like. Keep the search and user-fetch crawlers allowed if you want to be cited in AI answers.
The crawlers, by purpose
| robots.txt token | Company | Purpose | If you block it |
|---|---|---|---|
OAI-SearchBot | OpenAI | Search index | Pages can't appear as sources in ChatGPT search |
ChatGPT-User | OpenAI | User fetch | ChatGPT can't open the page when a user asks about it |
GPTBot | OpenAI | Training | Not used to train OpenAI models |
Claude-SearchBot | Anthropic | Search index | Pages can't appear in Claude's search results |
Claude-User | Anthropic | User fetch | Claude can't fetch the page for a user |
ClaudeBot | Anthropic | Training | Not used to train Anthropic models |
PerplexityBot | Perplexity | Search index | Pages can't be indexed and cited by Perplexity |
Perplexity-User | Perplexity | User fetch | Perplexity can't fetch the page for a user's question |
Googlebot | Search, AI Overviews, AI Mode | Gone from Google Search entirely | |
Google-Extended | Gemini training & grounding (robots.txt token only) | No effect on Search or AI Overviews | |
Bingbot | Microsoft | Bing index, used by Copilot | Gone from Bing and Copilot answers |
Applebot-Extended | Apple | AI training (robots.txt token only) | Not used to train Apple's models. Applebot itself still indexes for Siri/Spotlight |
DuckAssistBot | DuckDuckGo | AI answers | Not used in DuckDuckGo's AI answers |
Meta-ExternalAgent | Meta | Training | Not used to train Meta's models |
CCBot | Common Crawl | Open dataset (widely used for training) | Excluded from future Common Crawl snapshots |
Bytespider | ByteDance | Training | Not used to train ByteDance's models |
Companies add and rename crawlers. Check each vendor's crawler documentation before relying on a list, including this one.
Template 1: allow AI search, block AI training
Prefer to click? The AI robots.txt generator builds this for you.
The most common choice for businesses that want to be cited but don't want to contribute training data:
# AI training: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
Disallow: /
# AI search and user fetches: yes (same exclusions as everyone else)
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
Disallow: /admin/
Disallow: /cart/
# Everyone else
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlUser-agent: * group completely. That's why the search-bot group repeats your normal Disallow lines. If you write only Allow: / there, those bots can crawl your admin and cart paths too.Template 2: allow everything
No AI-specific rules needed. A robots.txt with only your normal exclusions allows every crawler:
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlTemplate 3: block all AI crawlers
If you list both search and training crawlers, you opt out of being cited too. Make sure that's what you want:
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
Disallow: /Don't add Googlebot or Bingbot to that list unless you also want to leave regular search.
robots.txt isn't the only switch
- CDN and firewall rules can block AI crawlers regardless of robots.txt (for example, Cloudflare's "Block AI bots" setting or its managed robots.txt). Check them when your robots.txt looks right but you're never cited.
- noindex keeps pages out of AI search as well as regular search.
- JavaScript-only content is invisible to most AI crawlers, which read the raw HTML. More in the technical GEO checklist.
Check what your robots.txt actually does
Matching rules are subtle: most specific group, longest matching path, Allow wins ties. Test a URL against each AI crawler with the AI visibility checker (all bots at once, with a suggested fix) or the robots.txt tester (one bot, the exact deciding line).