XML sitemap broken after a deploy? The 7 usual causes and how to check

6 min read

A broken sitemap rarely causes an instant traffic drop. That's what makes it dangerous: new and updated pages get discovered more slowly, Search Console starts reporting "Couldn't fetch", and nobody connects it to the release from two weeks ago. Here's what usually breaks and how to check in a couple of minutes.

The 7 usual causes

1. The route moved or disappeared

Framework upgrades and new routing setups often change how /sitemap.xml is generated. The file that a plugin or build step produced isn't produced anymore, and the URL returns 404 (or worse, your app's HTML 404 page with a 200 status).

2. HTML served instead of XML

Single-page apps commonly serve index.html for every unknown path. If the sitemap stops being generated, /sitemap.xml returns your homepage HTML with status 200. Crawlers can't parse it, and monitoring that only checks status codes says everything's fine.

3. Half the URLs vanished

A sitemap generator that reads from the database or CMS API can silently return a partial list: a pagination bug, a changed query, a content type left out, a timeout during build. The file looks valid; it just lists 400 URLs instead of 4,000.

4. Staging or wrong-host URLs inside

The sitemap is built from a configured base URL. Build it with the staging value and every <loc> points at staging.example.com, the same root cause as canonicals pointing to staging.

5. robots.txt points to the old location

The Sitemap: line in robots.txt still names the old path or an old domain after a migration. Search engines that discover sitemaps through robots.txt keep fetching the dead URL.

6. Blocked or challenged at the edge

A new bot-protection or firewall rule returns 403 or a JavaScript challenge to crawlers requesting XML files, while browsers sail through.

7. Too big or badly compressed

A sitemap file may hold at most 50,000 URLs and 50 MB uncompressed. Growing past that without switching to a sitemap index, or serving a .xml.gz with the wrong headers, makes it unreadable.

Check it in two minutes

1. Status and type: curl -sI https://example.com/sitemap.xml should return 200 and an XML content type (application/xml or text/xml), not text/html.
2. Is it really a sitemap? curl -s https://example.com/sitemap.xml | head -c 300 should start with <?xml and contain <urlset or <sitemapindex.
3. Count the URLs: curl -s https://example.com/sitemap.xml | grep -o "<loc>" | wc -l, and compare with how many indexable pages you have.
4. Check the host: curl -s https://example.com/sitemap.xml | grep -o "<loc>[^<]*" | head. Every URL should be on your production host and protocol.
5. Check robots.txt: the Sitemap: line should point to the current file.
6. Check Search Console: Indexing → Sitemaps shows the last read date and status for each submitted sitemap.

What happens when it breaks

Google doesn't deindex pages because a sitemap disappears; pages it already knows stay indexed and are still discovered through links. What you lose is speed and coverage: new pages, updated pages and deep pages with few internal links are found later, or not at all. For news, ecommerce and large content sites, that's real traffic.

It matters for AI search, too: Bing (which Microsoft Copilot relies on) and other crawlers use sitemaps and IndexNow to find fresh content.

Fix and prevent

• Generate the sitemap from the same source of truth as your pages, and make the build fail if it comes out empty or much smaller than last time.
• Return a real 404 (not your HTML app shell) for paths that don't exist.
• Keep Sitemap: lines in robots.txt current, and resubmit in Search Console and Bing Webmaster Tools after moving it.
• Monitor it: a daily check that the sitemap still parses and hasn't lost most of its URLs catches all seven causes above.