XML Sitemap

The format is a protocol. A sitemap opens with a urlset tag, holds a url entry for each page, and gives each entry a loc tag with the address; every other tag is optional, the file must be UTF-8, and all its URLs must come from one host^[1]. One file may list at most 50,000 URLs and be no larger than 50 MB uncompressed, and a larger site lists several sitemaps in a sitemap index file^[1].

What it does is help discovery. Google says a sitemap helps search engines find URLs on a site but does not guarantee that every item in it will be crawled and indexed^[2]. It adds the most on a large site, where it is harder to link every page, on a new site with few links from outside and on one with a lot of video or image content^[2]. A site of about 500 pages or fewer that is comprehensively linked from its home page may not need one^[2].

It is also a statement of intent. A sitemap tells search engines which pages the site thinks are important^[2], and Google advises listing the URLs the site wants to see in its search results, which are the canonical ones, and listing only the preferred address when one page has several^[3]. A URL that redirects, answers with an error or carries a noindex rule contradicts that statement, and Google’s Sitemaps report suggests replacing redirect URLs with the pages that should actually be crawled^[4].

A sitemap can be found in more than one way. A site can list its sitemap in its robots.txt file, the file that also holds its crawl rules^[4]. In QueryRev’s study, of the 6,294 sites whose robots.txt answer could be read, 5,637 had a file and 657 had none^[5].

Google’s Sitemaps report shows each submitted sitemap with its status (success, errors or could not fetch), the date it was last read and the number of page URLs discovered in it. Discovery is no guarantee that a URL has been or will be crawled or indexed, and the Page indexing report can be filtered by sitemap to see how many of its URLs are indexed^[4]. Only sitemaps submitted through the report or its API are listed there.

Several sitemaps are easier to read than one. Google notes that submitting several, for example one per section of the site, makes it possible to track the search performance of each in Search Console^[3]. A listed-to-indexed ratio per sitemap shows where the gap is.

Check the file against the site. Crawl every listed URL and record its status code, its canonical and whether it can be indexed. Then crawl the site from its home page and list the canonical pages that are linked but missing from the file, and the pages that are listed but linked from nowhere, an orphan page that only the sitemap vouches for.

Check the dates. Google uses a lastmod value if it is consistently and verifiably accurate, as when it matches the last modification of the page^[3]. Compare each lastmod with the page’s own last significant change.

Sample data: an invented site, to show the method.

A sample shop’s sitemap lists three URLs. The first is a canonical page that answers 200 and can be indexed. The second answers with a 301 redirect to the first. The third is a cart step that carries noindex.

The file tells search engines that the shop wants all three crawled. The redirect points to a page already listed, and the cart page says on its own that it should stay out of the index. A crawl of the three URLs finds both conflicts in a minute, and the fix is two deletions. The listed-to-indexed ratio of the file is judged only after that, because it counted two URLs that could never be indexed.