Accidental noindex: how one deploy wipes your traffic (and how to catch it)

7 min read

Most traffic disasters don't come from algorithm updates. They come from a single line that was right on staging and wrong in production. A noindex tag, a Disallow: / in robots.txt, or a canonical pointing at staging.example.com. Nobody notices for days, because the site still loads fine for humans.

This guide covers how it happens, how to find it in minutes, and how to recover.

The four ways a deploy can hide your site from Google

1. A noindex meta tag

Staging sites are (rightly) kept out of search with:

<meta name="robots" content="noindex, nofollow">

If that tag is driven by an environment variable, a config file or a CMS setting, it only takes one wrong value at release time for it to ship to production. WordPress's "Discourage search engines from indexing this site" checkbox is the classic version of this: it's ticked on staging, the database gets copied to production, and the checkbox comes with it.

2. An X-Robots-Tag header

The same instruction can be sent as an HTTP header instead of HTML:

X-Robots-Tag: noindex

This one is nastier, because it's invisible in the page source. It's usually added by a web server rule, a CDN transform or an edge function, and it can apply to every URL at once. If you only "view source" to check, you'll miss it. (More on that in X-Robots-Tag: the noindex you can't see in view-source.)

3. robots.txt blocking the crawler

A staging robots.txt typically says:

User-agent: *
Disallow: /

Ship that to production and Googlebot stops fetching your pages. Blocked pages can linger in the index for a while (often shown without a description), but they stop being refreshed, and any fix you make on the page, including removing a noindex, is never seen while the block is in place. Fixing robots.txt comes first.

4. A canonical pointing somewhere else

<link rel="canonical" href="https://staging.example.com/pricing">

A canonical tells Google which URL is the "real" one. Point it at a staging host, an http:// version, or the wrong page, and you're asking Google to consolidate your rankings somewhere else. Google treats canonicals as a strong hint rather than an order, which makes the damage slower and harder to spot, not smaller. (Deep dive: canonical tags pointing to staging.)

How to check a page in under a minute

For any important URL, look at four things:

1. Meta robots in the HTML — search the source for name="robots" and name="googlebot".
2. The response headers — look for X-Robots-Tag:

   curl -sI https://example.com/pricing | grep -i x-robots-tag

3. robots.txt rules for that path, for Googlebot specifically (Google uses the most specific user-agent group, and the longest matching rule wins). Our robots.txt tester shows the exact deciding line.
4. The canonical — it should usually be the page's own final URL, on your production host.

Or paste the URL into our free SEO config check: it does all four (plus redirects, title, SSL and domain expiry) the way a crawler sees the page, without running JavaScript.

Recovering after it happened

1. Fix robots.txt first if it blocks anything important, so Google can recrawl.
2. Remove the noindex from HTML and headers. Check both; fixing one and missing the other is common.
3. Fix canonicals to point at the production URLs.
4. Request recrawling in Google Search Console: use URL Inspection → Request indexing for your key pages, and resubmit your sitemap.
5. Watch the Pages report in Search Console. "Excluded by 'noindex' tag" and "Blocked by robots.txt" should fall over the following days to weeks, depending on how often Google crawls you.

Rankings usually come back once pages are reindexed, but it isn't instant, and the longer a page was out, the longer the climb back. The cheapest fix is catching it within the hour.

Preventing it

• Never control indexing with a value that's copied between environments. Decide "index or not" from the hostname at runtime, or protect staging with authentication instead of noindex.
• Keep staging robots.txt out of the build that goes to production.
• Add a release check that fetches a few key URLs after each deploy and fails if it finds noindex, a blocking robots rule, or an off-host canonical.
• Migrating or redesigning? Use the site migration SEO checklist.
• Monitor continuously. Releases aren't the only cause: CMS plugin updates, CDN rule changes and well-meaning colleagues change these settings too.