zh
English Español Deutsch Français Italiano Português (Brasil) Русский 中文 日本語
提交帖子
前往博客
Kevin Indig 已更新

XML Sitemaps Guide: The Best Tricks, Tips & Sitemap Generator Tools

73
Wow-Score
Wow-Score 表示博客帖子的参与度,是根据用户的主动阅读时间、滚动速度和文章长度之间的相关度计算得出。
此帖子为 English
Kevin Indig 已更新
此帖子为 English
XML Sitemaps Guide: The Best Tricks, Tips & Sitemap Generator Tools

In my first weeks at Atlassian, I was seeking for a quick-win to get a stronger foothold within the company and evangelize SEO. So, I audited our main site and noticed it didn’t have an XML sitemap. What an easy win!

I went to the devs and asked them to activate it in the CMS. To my surprise, they told me that it wasn’t possible; I was baffled.

After some thinking, I remembered that Screaming Frog had an XML sitemap function, so I scraped the site and uploaded the crawl as an XML sitemap. Google ate it within a few seconds, and we saw a noticeable impact on our traffic in the following days.

The moral of the story is that XML sitemaps are important and sometimes underrated.

Here is everything I am going to cover in this article

What XML Sitemaps Are and Why You Need to Have One

XML sitemaps are digital maps that help Google discover important pages on your site and how often they are being updated.

Google states on its help center page:

A sitemap tells the crawler which files you think are important in your site, and also provides valuable information about these files: for example, for pages, when the page was last updated, how often the page is changed, and any alternate language versions of a page.

According to Gary Illyes, XML sitemaps are the second most important source of URLs to be crawled by Googlebot after hyperlinks and previously discovered URLs. That’s massive and shouldn’t be underestimated!

Google started using XML sitemaps in 2005 and shortly after was joined by search engines like MSN or Yahoo. Nowadays, they use them for even more than just URL discovery.

Every website should have an XML sitemap. They are especially important for:

  • Large sites

  • New sites

  • Sites with lots of orphaned pages

  • Sites that use lots of images and videos

Whereas the robots.txt helps you to exclude parts of your site from being ranked in search engines, XML sitemaps do the opposite. They help search engines discover new pages – even when they are not linked from the main site.

Sitemaps come in XML format that Google can quickly parse to find new URLs. XML — eXtensible Markup Language — is lightweight and portable between devices and was made to store data.

The easiest way for you to check if your site has a sitemap is to look in Google Search Console or in Bing Webmaster Tools under “sitemaps.” Most search engines, such as Google or Bing, look for the “Sitemap: <sitemap_location>” entry (or entries) in your site’s robots.txt file. Alternatively, you can also ping your sitemap directly to Google, Baidu, Bing, and Yandex.

xml sitemap in Bing Webmaster Tools XML sitemaps in the Bing Webmaster Tools. 1: Sitemaps report. 2: Adding new sitemap paths. 3: existing sitemaps Bing found.

XML sitemap in Google Search Console XML sitemaps in Google Search Console. 1: Sitemaps report. 2: Adding new sitemap paths. 3: existing sitemaps Google found.

HTML vs. XML Sitemaps 

There are two types of sitemaps: HTML and XML. What is the difference?

1. You will notice the format.

HTML is obviously different from XML. But that implies even more: while HTML sitemaps are visible to site users, XML sitemaps are feeds for search engines.

You could argue that HTML sitemaps are also created for search engines, but while they can be valuable to users, XML sitemaps cannot.

2. They serve the same purpose but in different ways.

Both help search engines discover new URLs, whether pages, videos, or images. 

XML sitemaps are custom feeds that help search engines understand the priority of URLs to crawl, how often they change, and which new ones were added to the site. That is especially helpful for search engine schedulers because they can better estimate when and how often to recrawl a URL.

HTML sitemaps also help search engines discover new URLs but through the discovery of links they follow. That means HTML sitemaps can only be an effective URL discovery tool if they are being crawled and if the links are followed. You can understand this by looking at your log files.

3. They have different side-benefits.

XML sitemaps have meta-attributes like <changefreq> or <lastmod> to indicate how the state of URL changes. They can also carry extensions for videos, images, and news.

HTML sitemaps distribute PageRank throughout a site, and that is what they are nowadays mainly used for, aside from the navigational value for users. Since HTML sitemaps are often linked in the footer of a site, they are usually linked from every page and might distribute that incoming PageRank to other pages with weaker internal linking.

Different Types of XML Sitemaps

Even though XML sitemaps can be submitted in RSS, mRSS, Atom 1.0 or text format, the “type” of a sitemap refers to its content or “media type”:

  • News

  • Video

  • Image

As I will further specify below, you can create sitemaps that contain only one specific media type or integrate them into your regular XML sitemap.

XML Sitemap Minimum Requirements

For your XML sitemaps to work optimally, you have to meet the standards. An XML sitemap should:

  • Contain only canonical URLs with a 200 status code.

  • Include up to 200K URLs per sitemap and up to 50K sitemaps per index sitemap.

  • Be referenced in the robots.txt.

  • BUTF-8 encoded.

  • Be compressed in .gz format.

  • Be no larger than 50mb or contain 50,000 URLs (whatever you hit first).

But there is more you can and should do to get the most out of XML sitemaps. You can signal to Google which URLs are important by including only important pages in XML sitemaps, and by updating it often.

Most CMSs have a function to automatically update sitemaps when a new URL is created or an existing page changes. For Google, the update frequency of the sitemap itself and the lastmod tag of pages can be a signal of freshness. Whether that is important for its ranking depends on the page and the context.

Think of an XML sitemap as a city map for tourists with the city being your website and the tourist being Google — you want to make sure to only include the important buildings, not every address. That is why unimportant pages shouldn’t be included; examples are pages like your privacy policy or about us page. While these pages should be indexed, they don’t need to be crawled often and are not important when we are talking about SEO.

XML Sitemap Tips for Large Sites

There is more you can do to elevate your sitemap game, beyond meeting the standard requirements.

Large sites like news publishers, for example, should make use of index sitemaps, which contain (up to 50,000) normal sitemaps, and should also not be heavier than 50mb. They are like the XML sitemap mothership that carries lots of smaller sitemaps. Large sites need them because they can’t fit into a single sitemap. You shouldn’t try to fit everything into a single sitemap, anyway.

You can make the most out of these sitemaps by structuring them either per page type or topic. In practice, you would create dedicated XML sitemaps per subdirectory or page template to get an understanding of technical and indexing problems with your site.

GSC submitted sitemap xml

There are specialized XML sitemaps for specific purposes. Sites that operate heavily around rich media (think: Pinterest or YouTube) benefit a lot from image or video sitemaps. Publishers should have news sitemaps.

Image sitemaps increase your site’s chance to be found in Google image search. You don’t have to have a dedicated image sitemap; you can also use image extensions in your regular sitemap.

This is what image extensions look like (XML specifications):

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
<url>
<loc>http://example.com/sample.html</loc>
<image:image>
<image:loc>http://example.com/image.jpg</image:loc>
</image:image>
<image:image>
<image:loc>http://example.com/photo.jpg</image:loc>
</image:image>
</url>
</urlset>

Video sitemaps function after the same principle: either create a dedicated sitemap or add extensions to your regular one:

<url>
<loc>https://example.com/mypage</loc>
<video> ... information about video 1 ... </video>
</url>

But be careful with the meta-data you add to video sitemaps or extensions.

Google states, “Google might use text on the video landing page rather than the text you supply in your sitemap if the page text is deemed more useful than the information in the sitemap.” They are speaking about the text delivered through the description. Besides a description, you can feed Google a thumbnail, video length, rating, family-friendliness, and more (full list of video XML sitemap meta-data). For sites that heavily use video, this certainly makes sense. For all others, it is relatively optional.

News sitemaps are different in that you should always have a separate news XML sitemap. Google doesn’t recommend (or offer) extensions in this case. News sitemaps help Google discover and rank new articles, which is especially challenging in the publishing industry because it produces a lot of content. Even though Google states that publishers with news sitemaps are not favored, it does help to get hot news ranking in Google News faster.

News sitemaps have special requirements:

  • Include articles not older than 2 days.

  • Don’t add more than 1000 new entries to an existing sitemap at a time.

  • Update existing sitemaps for article updates.

You can also use XML sitemaps to define and indicate certain meta-tags for Google. One example is hreflang; you can add as an extension to a sitemap (full guidelines):

<url>
<loc>http://www.example.com/english/page.html</loc>
<xhtml:link
 rel="alternate"
 hreflang="de"
 href="http://www.example.com/deutsch/page.html"/>
<xhtml:link
 rel="alternate"
 hreflang="de-ch"
 href="http://www.example.com/schweiz-deutsch/page.html"/>
<xhtml:link
 rel="alternate"
 hreflang="en"
 href="http://www.example.com/english/page.html"/>
</url>

Google ignores the priority attribute in XML sitemaps but does pay attention to lastmod, according to John Mueller. Google determines the priority of your pages itself, probably by popularity and authority. Lastmod, however, is a tag that indicates when the URL has changed the last time, which is really interesting to Google.

Also, you don’t need to add XML sitemaps for AMP URLs, according to John Mueller.

XML Sitemap Best and Worst Practices

At Atlassian, we solved the missing XLM sitemap functionality of our CMS with a 3rd party XML sitemap provider, and it worked just fine.

3rd party xml sitemap provider example

Even though the format is text-based instead of XML, it works.

The New York Times references its sitemaps in the robots.txt and separates formats like videos or news. It goes even a step further and has sitemaps for specific categories, such as cooking or elections.

It makes sense to have dedicated XML sitemaps to timely events as a publisher because you need to understand how fast Google picks the content up and if everything can be indexed without problems.

Events in sitemaps


Walmart has a similar split by categories that makes a lot of sense for e-commerce sites. It has Master XML sitemaps for topics and categories.

Example of a Master XML sitemap for topics and categories


As you can see in the screenshot below, the topic split allows Walmart to see how Google indexes different areas of the site like fashion or entertainment.

sitemap topic split example

If you have a site that is split into topic, categories, or both, creating specific XML sitemaps for each is recommendable. There is no known disadvantage of having the same URLs in different sitemaps.

SEMrush Tip: With the SEMrush Site Audit tool, you can audit any website and check for six specific issues related to XML sitemaps. The tool will first check for if a sitemap.xml is present or not, and then it will look for formatting errors, incorrect pages in the sitemap, and other issues that could be impacting the clarity of your sitemap. 

XML Sitemap Tools and Generators

Most content management systems come with prepackaged functions that allow you to create an XML sitemap automatically. But some don’t, and in this case, you need a third-party tool.

Best Sitemap Generators

These are my personal picks for XML sitemap generators.

Name

Price

Limit

Features

Free trial

Slickplan

$8.99/month

n/a

  • Drag and drop builder
  • Custom page type inclusion
  • Import text file
  • Cloning
  • Batch editing
  • Highly customizable
  • User permissions
  • Custom branding

30 days

Dynomapper

$40/month

200K URLs per crawl

  • Monitor URLs in sitemaps in Google Analytics
  • Highly customizable
  • Custom page type inclusion
  • Workflow management
  • URL tagging
  • Sitemap filtering
  • User permissions
  • Custom branding

14 days

Writemaps

$14.99/month

n/a

  • Customizable
  • Custom groups
  • Drag and drop builder

3 sitemaps free

Screaming Frog

Free to 500 URLs

£149.00/year

n/a

  • Not made for XML sitemaps but good workaround for technical restrictions

n/a

Powermapper

$49/month

15K pages

  • Not made for XML sitemaps but can export a feed into XML format

30 days

XML Sitemaps

$4.99 for 1K pages

$189.99 for 1.5m pages

1,5m pages

  • Image and video sitemaps
  • Email notifications
  • Mobile app
  • Detects broken links

Free for 500 pages

Best Wordpress Plugins for Sitemaps

Name

Price

Limit

Features

Ratings

Simple Wp Sitemap

free

n/a

  • HTML and XML sitemap
  • Dynamic sitemaps

4.4/5

33 reviews

Google Sitemap by BestWebSoft

free

n/a

  • Hreflang support
  • Customizable

4.3/5

112

Google XML Sitemaps

free

n/a

  • Dynamic sitemaps
  • Customizable

4.9/5

2090 reviews

Yoast SEO

free (premium available)

n/a

  • Basic, dynamic sitemap

4.9/5

26,745 reviews

WordPress XML Sitemap Plugin

free

n/a

  • Highly customizable

n/a

All in One SEO Pack

free

n/a

  • Basic, dynamic sitemap

4.4/5

449 reviews

XML Sitemap & Google News

free

n/a

  • Basic, dynamic sitemap
  • Customizable
  • Updates lastmod automatically

4.4/5

59 reviews

Do you have any XML sitemap tips or tools? Let us know in the comments.

Get a free 7-day trial

Start working on your online visibility

Please specify a valid domain, e.g., www.example.com

Kevin Indig
专栏作家

SEMrush 专栏作家是指定期为我们的社区做出贡献的数字营销资深作者。

发送反馈
您的反馈必须包含至少 3 个单词(10 个字符)。

我们只会使用您的电子邮件,以回答您的反馈。 隐私政策

感谢您的反馈!