Is your site invisible to ChatGPT? A technical GEO checklist

7 min read

Generative Engine Optimization (GEO) is usually discussed as a content problem: write clear answers, earn mentions, be the source worth quoting. All true, but it skips a step. An AI answer engine can only cite a page it was allowed to fetch and could actually read. On a surprising number of sites, one of those two fails, and nobody notices because the site looks fine in a browser.

This is the technical half of GEO: the checks that decide whether you're even in the running.

1. Know which AI crawlers matter

AI companies run several crawlers with different jobs, and robots.txt treats each one separately:

The important distinction: blocking a training crawler doesn't remove you from AI search. You can block GPTBot and still be cited by ChatGPT search, as long as OAI-SearchBot is allowed. Many sites copied a "block all AI bots" list in 2023–24 that included the search crawlers too, and opted out of AI search without meaning to.

And note Google: there is no separate AI Overviews crawler. If Googlebot can index the page, it's eligible. Google-Extended only covers Gemini training and grounding.

2. Check robots.txt for each of them

A typical accidental block looks like this:

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: ClaudeBot
Disallow: /

If you want training blocked but citations allowed, split it:

# No training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

# AI search and user fetches welcome
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

Remember the matching rules: a crawler uses the most specific group that names it and ignores User-agent: * entirely when one exists. A bot that's named in a group with only a few Disallows doesn't inherit your general rules. Test individual URLs with our robots.txt tester. Full list of crawler tokens and more templates: AI crawler user agents & robots.txt templates.

3. Check your CDN and firewall

robots.txt is a polite request. CDNs and WAFs enforce. Cloudflare, for example, has a one-click "Block AI bots" setting and a managed robots.txt, and other providers and security plugins offer similar switches. These are often turned on during a scraping scare and forgotten. The result is a site whose robots.txt says "welcome" while the edge returns 403 to every AI crawler.

To check, look at your CDN's bot settings and firewall events filtered by user agent (step by step for Cloudflare: is Cloudflare blocking AI crawlers?). Our AI visibility checker also sends requests that identify as the main AI crawlers and compares the responses with a normal request.

4. Make sure the content exists without JavaScript

Googlebot renders JavaScript. Most AI crawlers don't: they read the HTML your server sends. A client-rendered app sends something like this:

<body>
  <div id="root"></div>
  <script src="/assets/app.js"></script>
</body>

To an AI crawler, that page is empty. Server-side rendering or static generation (Next.js, Nuxt, Astro, SvelteKit and similar all support it) fixes this. A quick test is curl -s https://example.com/page | grep -c "a sentence from your page". If that prints 0, AI crawlers can't see it either.

5. Don't forget the basics still apply

• noindex in a meta tag or X-Robots-Tag header keeps pages out of AI search too (the header is easy to miss).
• Canonicals pointing elsewhere tell every engine to credit another URL (common causes).
• Bing indexing matters more than it used to: Copilot runs on it, and other AI search products use it for retrieval. Verify your site in Bing Webmaster Tools and submit your sitemap there.

6. Make pages easy to quote

Once access is solved, structure decides whether a passage gets lifted into an answer:

• One clear H1 and question-style H2s, each followed by a direct answer in the first sentence or two.
• Structured data (JSON-LD) that states the facts plainly: Organization, Article with dates, Product with price, FAQPage where it fits.
• Visible dates and dateModified for content where freshness matters.
• Specifics: numbers, names, definitions and comparisons are what answer engines quote.
• llms.txt is optional. It's a proposed standard with a Markdown summary of your site for language models. It's cheap to add, but no major AI search engine has confirmed using it, so do it after everything above (more on whether llms.txt matters).

Quick audit

1. Run a key URL through the AI visibility checker.
2. Fix any AI search crawler blocks in robots.txt. Keep training blocks if you want them.
3. Check CDN/WAF bot settings.
4. Confirm the main content is in the raw HTML.
5. Add JSON-LD and tighten headings on your most important pages.