AI robots.txt generator
Decide which AI crawlers may use your site. The usual choice: block training, keep AI search, so ChatGPT, Perplexity and Claude can still cite you. Runs in your browser; nothing is sent anywhere.
| Allow | Crawler | Purpose |
|---|
One per line, e.g. admin, cart, internal search.
Upload it as /robots.txt at your site root, then confirm with the AI visibility checker. robots.txt doesn't override CDN rules: if your CDN blocks AI bots, change it there too.
Why not just block every AI bot?
Because AI companies use different crawlers for training and for search. Blocking GPTBot keeps your content out of OpenAI's training data; blocking OAI-SearchBot removes you from ChatGPT search results. Most businesses want the first without the second.
Why does the generated file only name the blocked bots?
A crawler uses the most specific group that names it and otherwise falls back to User-agent: *. Allowed AI crawlers therefore follow your normal rules automatically. Naming them in a group of their own would make them ignore your * exclusions unless you repeated them.
What about Google AI Overviews?
There's no separate crawler: AI Overviews and AI Mode use Googlebot. Google-Extended only controls Gemini training and grounding. Blocking it doesn't affect Search.
Do AI crawlers respect robots.txt?
The major ones (OpenAI, Anthropic, Google, Perplexity, Apple, Common Crawl) document that they do. For crawlers that don't, you need a CDN or firewall rule. More detail in AI crawler user agents & robots.txt templates.