Your XML Sitemap Is Lying About Your Site. The 15-Minute Cleanup

Someone recently told me, “My site has 40 pages, but Google only seems to know about 22.”

The site was built quickly with an AI-assisted website builder. The owner had launched several new pages, renamed old ones, removed temporary pages, and changed the domain from www to the root domain.

The XML sitemap still contained almost every old URL.

That is how a sitemap starts lying.

An XML sitemap should be a clean list of the canonical, public, indexable pages you want search engines to discover. It should not be a historical record of every page your site has ever had.

This 10 to 15 minute cleanup helps you fix sitemap problems across three areas:

• Traditional SEO: Help Google crawl and index the right pages.
• AI search visibility: Give answer engines clearer signals about which pages matter.
• Website security: Avoid exposing backup files, admin paths, and forgotten public URLs.

Google explains that sitemaps help search engines discover pages and understand important site content. They are especially useful for new, large, or frequently changing websites. AI-built websites often qualify because their URL structures change quickly during development.

What makes an XML sitemap inaccurate?

A sitemap is sending bad signals when it includes URLs that are not suitable for indexing.

Sitemap problem	What it tells search engines	Correct action
404 or deleted URL	“This important page exists,” but the server says it does not	Remove it or restore the page
Redirecting URL	“Use this URL,” but it sends visitors elsewhere	List the final destination
noindex page	“Index this page,” while the page says not to index it	Remove it or remove noindex
Non-canonical URL	“This version matters,” while another URL is canonical	List the canonical URL only
Duplicate URL	Multiple URLs represent the same content	Consolidate and keep one URL
Stale renamed page	An old page still appears important	Replace it with the new URL
HTTP or www mismatch	The sitemap points to a different site version	Use the live HTTPS host consistently
Blocked sitemap	Crawlers cannot retrieve the file	Make the sitemap publicly accessible
Authenticated sitemap	Search engines cannot access the sitemap without login	Move it to a public URL

❌ Bad sitemap entry

http://www.example.com/old-pricing

The URL redirects to:

https://example.com/pricing

✅ Clean sitemap entry

https://example.com/pricing

The same rule applies to pages built with no-code tools, AI website generators, WordPress, Shopify, Webflow, and custom code.

Why this matters for SEO and AI search

A clean sitemap does not guarantee rankings. It does make your site easier to understand.

For traditional SEO, your sitemap should support the same structure as your internal links, canonical tags, robots.txt rules, and indexability settings.

For AI search, the same clarity matters. Search systems and answer engines need to identify:

• Which pages are public
• Which URLs are current
• Which pages describe your products or services
• Which content is authoritative
• Which URLs should be cited instead of old duplicates

An accurate XML sitemap combined with a useful llms.txt file gives AI crawlers a clearer map of your website. The sitemap identifies your valid public URLs. The llms.txt file can highlight the pages you want AI systems to understand and prioritize.

It is not a guarantee that ChatGPT, Perplexity, or another answer engine will cite your site. It is a way to reduce ambiguity.

The 15-minute XML sitemap cleanup

1. Find the sitemap and confirm the correct domain

Time: 1 minute

Open these common locations:

https://example.com/sitemap.xml
https://example.com/sitemap_index.xml
https://example.com/robots.txt

Your robots.txt file should usually contain a public sitemap directive such as:

Sitemap: https://example.com/sitemap.xml

Check all four details:

• Does the URL use https://?
• Does it use the correct www or non-www version?
• Does it load without a login?
• Does it return a normal XML response instead of an error page?

A sitemap behind authentication is not useful to search engines. If the sitemap requires a password, move it to a publicly accessible location.

Use the free Robots.txt Generator to create or correct the file. It requires no login, email, or paywall.

2. Open the sitemap report in Google Search Console

Time: 2 minutes

In Google Search Console:

1. Open the correct property.
2. Go to Indexing.
3. Select Sitemaps.
4. Click your submitted sitemap.
5. Review the discovered URL count.
6. Open the related page indexing information when available.

Look for a large difference between URLs submitted in the sitemap and URLs Google considers indexed.

A difference does not always mean something is broken. Some pages may be new, duplicate, low quality, or intentionally excluded. However, a large unexplained gap deserves investigation.

Also check whether the sitemap itself shows:

• Couldn’t fetch
• Sitemap is blocked
• Invalid URL
• HTTP error
• Incorrect file format
• URLs outside the property

3. Use the Pages report to find indexability failures

Time: 4 minutes

Go to Indexing > Pages in Google Search Console.

Review the excluded and error categories that commonly affect sitemap URLs:

• Not found, 404
• Page with redirect
• Excluded by noindex
• Duplicate without user-selected canonical
• Alternate page with proper canonical
• Crawled, currently not indexed
• Discovered, currently not indexed
• Blocked by robots.txt
• Server errors

Open a sample from each relevant category.

Your goal is not to force every URL into Google. Your goal is to ensure that every URL in the sitemap is intentionally included.

Use this decision rule:

• Keep it if the URL returns 200 OK, is indexable, is canonical, and belongs in search.
• Fix it if the page should exist but has a technical problem.
• Remove it if it is deleted, private, duplicated, temporary, redirected, or intentionally excluded.

4. Compare sitemap URLs with your real pages

Time: 3 minutes

Create two simple lists:

List A: Sitemap URLs

Copy the URLs from your sitemap or sitemap index.

List B: URLs reported by Search Console

Use the Pages report, URL Inspection, or an export from your SEO workflow.

Now compare them by pattern.

This is where the free Search Console Regex Library helps. It contains ready-to-use RE2 patterns for filtering page URLs and queries inside Google Search Console.

Use URL filters to isolate likely problem groups:

(?i)(old|archive|test|staging|backup|admin|login)

(?i)(\?|utm_|amp|print|preview)

(?i)(http://|www\.)

The library is useful for segmenting URL patterns and finding groups that should not be treated the same way. It does not replace a direct sitemap crawl or a manual comparison. Use the filtered results to identify:

• Pages in the sitemap but not indexed
• Indexed pages missing from the sitemap
• Old URL patterns
• Query-string duplicates
• Test or staging paths
• HTTP and www variants

5. Correct the file, not just Search Console

Time: 3 minutes

Do not simply delete a sitemap submission from Search Console and assume the problem is fixed. Update the source that generates the sitemap.

For an AI-built or no-code website, check:

• CMS sitemap settings
• Route configuration
• Static page generation
• Redirect rules
• Canonical tag settings
• Deployment environment variables
• Domain redirects
• Temporary pages left in the build

A clean sitemap should contain only URLs that meet all of these conditions:

200 OK
Publicly accessible
Canonical
Indexable
Useful
Correct HTTPS host

Remove:

• 404 URLs
• 301 and 302 URLs
• noindex URLs
• Duplicate URLs
• Filtered URLs
• Search result pages
• Preview URLs
• Login and account pages
• Staging pages
• Temporary campaign pages
• Old paths from renamed pages

Then resubmit the updated sitemap in Search Console.

6. Check the security angle

Time: 2 minutes

Your sitemap is also a public description of your website’s shape.

That makes it useful for legitimate crawlers, but it can also reveal paths you should never have left public.

Look for entries such as:

/backup/
/old-site/
/admin/
/test/
/staging/
/uploads/database.sql
/config/
/private-report/

Do not add sensitive files to a sitemap so that search engines can “find” them. Remove exposed files from the server, restrict private areas with authentication, and use the correct access controls.

Remember that robots.txt is not a security boundary. Google states that robots.txt controls crawling, but it does not reliably keep a private page out of search results. A blocked URL can still be discovered through external links.

Use password protection, proper server permissions, or remove the file entirely.

Run the free Vulnerability Check to look for exposed files, unsafe configurations, leaked credentials, and other website security issues. It requires no login, email, or paywall.

Connect the three files: sitemap, robots.txt, and llms.txt

Think of these files as serving different purposes:

File	Main job
XML sitemap	Identify current, canonical URLs you want crawled
robots.txt	Manage crawler access and point to the sitemap
llms.txt	Explain your important pages to AI crawlers and answer systems

Avoid contradictions.

❌ Contradictory setup

• Sitemap lists /old-guide
• /old-guide redirects to /guide
• Robots.txt blocks /guide
• llms.txt links to /old-guide

✅ Consistent setup

• Sitemap lists /guide
• /guide returns 200 OK
• /guide is canonical and indexable
• Robots.txt does not block /guide
• llms.txt identifies /guide as the current resource

Generate a clean llms.txt file with the free llms.txt Generator. You can also use the free Schema Markup Generator to add structured data that clarifies your organization, articles, services, and frequently asked questions.

Both tools are available without a login, email, or paywall.

Final checklist for your 15-minute cleanup

✅ Sitemap loads publicly
✅ Sitemap uses the correct HTTPS domain
✅ No HTTP or www mismatches
✅ No 404 URLs
✅ No redirecting URLs
✅ No noindex pages
✅ No duplicate or non-canonical URLs
✅ No stale renamed pages
✅ No query-string duplicates
✅ No private, admin, backup, or staging paths
✅ Robots.txt includes the correct sitemap location
✅ Important sitemap pages are internally linked
✅ llms.txt points to current, useful pages
✅ Schema markup describes the page accurately

After making changes, run a free SEO/AEO analyzer. It checks sitemap signals, robots.txt, structured data, technical SEO, on-page SEO, and answer engine visibility. The complete report is free with no signup, no email, and no paywall.

Your sitemap should not be a scrapbook of old experiments. Keep it short, current, public, and accurate.

That makes it more useful for Google, easier for AI search systems to interpret, and less likely to expose the hidden leftovers of a site built too quickly.