Is Cloudflare blocking ChatGPT and other AI crawlers on your site? How to check and fix it

6 min read

Cloudflare sits in front of a large share of the web and gives site owners several ways to stop AI crawlers. That's useful if you don't want your content used for training. But the same switches can also block the crawlers that ChatGPT search, Perplexity and Claude use to cite you, and some of them can be turned on without anyone on your team remembering it.

Cloudflare renames and reorganises its dashboard often. The setting names below describe what to look for; the exact menu path in your account may differ.

The four places Cloudflare can block AI crawlers

1. The "block AI bots / AI scrapers" setting

A one-click option in the bot settings that blocks known AI crawlers at the edge. It returns an error or a challenge regardless of what your robots.txt says. Depending on the version, it can cover only training crawlers or a broader list that includes search and user-triggered fetchers.

2. Default blocking on newer domains

Cloudflare has moved towards blocking AI crawlers by default for newly added domains, asking during setup how AI bots should be treated. If someone clicked through that screen, the choice may be "block".

3. Managed robots.txt

Cloudflare can add AI crawler rules to your robots.txt for you, prepending Disallow groups for crawlers like GPTBot, ClaudeBot and Google-Extended, sometimes with a Content-Signal line such as search=yes, ai-train=no. Check which bots it names: if search crawlers are in the list, you're opted out of AI citations too.

4. Your own WAF and bot rules

Custom firewall rules, rate limits, country blocks or "bot fight" modes written during a scraping incident can catch AI crawlers as collateral damage, especially rules matching user agents with words like "bot" or "GPT".

How to check

1. Read your live robots.txt (https://yoursite.com/robots.txt), not the file in your repo. Managed rules are added at the edge, so only the live version shows them. Look for OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot.
2. Check the AI crawler / bot settings for the zone: is AI bot blocking on, and which crawlers does it cover?
3. Look at the security events filtered to user agents containing OAI-SearchBot, ChatGPT-User, PerplexityBot or Claude. Blocks and challenges there are your answer.
4. Run an outside check. Our AI visibility checker reads the live robots.txt per AI crawler and sends requests that identify as AI crawlers to see if the edge treats them differently. Cloudflare verifies real bots by IP, so a pass there is a good sign but not proof. The security events are authoritative.

The setup most businesses want

Block AI training, allow AI search. In practice:

• If you use Cloudflare's AI bot blocking, make sure search and user-fetch crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User) are allowed. Many plans let you set crawler-by-crawler allow or block.
• In robots.txt, block training crawlers by name and leave search crawlers on your normal rules. The AI robots.txt generator writes this for you.
• Leave Googlebot and Bingbot alone. Google's AI Overviews use Googlebot, and Microsoft Copilot relies on Bing.

Keep it from changing silently

Bot settings get changed during incidents, plan changes and dashboard migrations, and the effect on AI visibility is invisible: there's no Search Console for ChatGPT. Watching your live robots.txt is the cheapest early warning.