X-Robots-Tag: the noindex you can't see in view-source

5 min read

When a page drops out of Google, the first thing most people do is open the page source and search for noindex. If it isn't there, they move on. But there's a second way to say "don't index this", and it never appears in the HTML at all: the X-Robots-Tag HTTP header.

What it is

X-Robots-Tag is a response header that accepts the same directives as the robots meta tag:

HTTP/2 200
content-type: text/html; charset=utf-8
x-robots-tag: noindex, nofollow

Google treats it exactly like <meta name="robots" content="noindex, nofollow">. It can also target one crawler (X-Robots-Tag: googlebot: noindex) and, unlike a meta tag, it works on files that have no HTML: PDFs, images, feeds.

If the header and the meta tag disagree, the more restrictive directive wins. A page with index, follow in its HTML and noindex in its headers is not indexed.

Where it sneaks in

• Web server config. A line like add_header X-Robots-Tag "noindex"; (nginx) or Header set X-Robots-Tag "noindex" (Apache) added for staging and left in a shared config file.
• CDN and edge rules. Transform rules, edge functions or middleware that add headers to "non-production" hostnames, where the hostname match is broader than intended.
• Framework defaults for preview deploys. Many hosting platforms add X-Robots-Tag: noindex to preview/branch URLs on purpose. That's correct until production is accidentally served from one of those URLs, or a custom domain is pointed at a preview.
• SEO plugins. Some plugins can send the header for whole sections (archives, attachments, search pages); a settings change can widen that.
• Files. Teams often noindex PDFs with a header rule like "all *.pdf", then wonder why their whitepaper isn't ranking.

How to check for it

From a terminal:

curl -sI https://example.com/pricing | grep -i x-robots-tag

No output means no header. A few things to keep in mind:

• Check the final URL after redirects (curl -sIL), and check every hop if you care about the redirect target specifically.
• Some setups only send it to certain user agents or from certain regions, so test with a crawler-like request too.
• In Chrome DevTools: Network tab → click the document request → Response Headers.
• In Google Search Console, URL Inspection shows "Indexing allowed? No: 'noindex' detected in 'X-Robots-Tag' http header" once Google has crawled it.

Our free SEO config check reads the header alongside the meta robots tag, robots.txt and canonical, and tells you which one is responsible.

Fixing it

1. Find which layer adds it: origin server, framework, CDN/edge, or plugin. Compare headers from the origin directly (bypassing the CDN) and through the CDN.
2. Remove it for production hostnames only. Keep it on staging and previews, but make the rule match those hostnames explicitly rather than "everything except production".
3. Request indexing for key URLs in Search Console once the header is gone.