and when to use HTML sitemaps in SEO and UX

John Mueller, a Google search advocate, has always minimized the importance of HTML sitemaps. He recently reiterated his dissatisfaction regarding HTML sitemaps.

“I have changed my mind about HTML sitemaps throughout the years. They should never be necessary.” Sites large and small should have clear navigational structures.

This is a slightly stronger position than the he has previously expressed:

“When it comes SEO, small sites should be easily crawlable (and if your CMS is common, it will almost always work fine). Large sites won’t be able to use sitemaps or normal cross-linking without consulting a crawler.

Mueller is correct. However, I don’t think we can just dropHTML sitemaps. Still, links and website usability are important.

I like also what Pedro Dias , a former Googler and SEO expert, said in response Mueller’s comments. It reflects the reality on the web.

“Unfortunately, it is not easy to modify the architecture of large and complex websites. This is a common situation. It can take many years to make changes, since it is dependent on multiple people. HTML sitemaps can be a temporary but welcome solution. They used to be something I resented, but I now see their value.

There is a compromise between HTML and XML sitemaps. SEO says “it depends”.

Learn below about sitemaps and the benefits they can bring to your website and users.

Sitemaps: What are they and how did they come to be?

Many websites were created organically in the early days.

Information architects didn’t plan websites according to UX best practice.

They began from scratch and added content as needed, often in unusable ways.

These were before social media, blogs and content management systems. You built a website using HTML.

Most cases required you to code the HTML manually in a text editor. You could use a confusing WYSIWYG editor such as Dreamweaver, or one of the many custom-built, cumbersome CMS tools. Website structures quickly become confusing, regardless of how they were constructed.

To make sense of the growing chaos, people added sitemaps onto their websites out of necessity. Sitemaps in plain HTML looked just like any other webpage. This was due to the fact that they had a map and not written content.

Later search engines introduced XML websitemaps to replace manual submission.

HTML sitemaps have been less popular for 25 years. Can we stop using HTML sitemaps like we stopped using webrings or guest books?

HTML vs. XML Sitemaps – The visible vs. invisible

It is important to note that HTML sitemaps are visible to site visitors, while XML sitemaps are not. The latter is primarily for search engines.

Search engines such as Google and Yahoo introduced the XML sitemaps standard at the end 2007 (then MSN).

They are not easily readable or displayed well by default. These sitemaps serve primarily to improve search engine indexing.

However, just because a resource appears on an XML sitemap doesn’t mean Google or Bing will index it.

Rob Watts, another SEO colleague added a tongue in cheek, first-hand account of the debate by saying

I had sites that did well for years without html or sitemaps. I took off my tin foil hat, and started using xml websitemaps. It was amazing how long it took for URLs to show up in search results. Although I didn’t notice any performance improvements, it did make some reports/analyses a little easier.

XML sitemaps are theoretically responsible for crawling and indexing. Do we need to be so focused on them and ignore visible or visually sitemaps instead?

Why should sitemaps be created only for Google?

Google spokespeople have repeatedly told us that websites should be created for users and not search engines over the years. Sitemaps are important for Google and other search engines.

This makes me wonder if other SEO strategies that show one thing to search bots but another for website visitors might be a better idea.

Google has said it, so we should listen. It’s not.

Google must have a reason for preferring XML to HTML sitemaps.

Sitemaps using XML are often automated and include all content that has been published online.

Numerous tools can create sitemaps on WordPress. Yoast can also create sitemaps for you.

WordPress has added XML support to its core in 2020. This means that you don’t need any external plugins.

XML is machine-readable while HTML is messy. HTML must work despite errors while XML can’t be broken if you forget a character.

The XML sitemap creator works almost entirely behind the scenes, without any human intervention.

Also, the majority of people won’t see your XML sitemap. They wouldn’t know where it is because it’s not linked anywhere on the site.

People make XML sitemaps work

There is always a middle ground. You don’t need to make XML sitemaps inaccessible.

Stylesheets can be used to format XML sitemaps, just as you would with HTML pages.

This technique has been in use for as long as XML has existed. There tutorials available to assist you with this.

It seemed for a time that XML would replace HTML, or it was a compromise known as XHTML.

This was not possible due to compatibility. Even though XHTML was simpler and less likely to make errors, it still worked. The website would break if an error occurred.

Instead, HTML5 was introduced. Browsers adopted sloppy coded websites to simplify publishing but at the cost of automatic crawling.

You have now created an XML sitemap which can be easily read by anyone without coding skills. It’s still just a list of the content that you have.

It is worth considering creating an accessible HTML sitemap for your website visitors.

Use visual or HTML sitemaps when appropriate

It is not either/or. You can:

• Instead of making your XML sitemap visible, decide to seperate visible and invisible sitemaps.
• For different reasons, create sitemaps that are both visible (HTML), and invisible (XML).
• You can have more than one HTML sitemap or XML sitemap.

It all depends on how your site looks and what your goals are.

These questions can be used as a guideline to help you make a decision.

• You want to quickly create an overview page for your website? You can use an HTML sitemap to help you get started!
• Are you looking to be indexed as quickly and deeply as possible? Make an XML sitemap.

However, there are exceptions and websites that do not meet the standard.

These are examples of HTML or visual sitemaps.

Sites that are hard-coded and legacy sites

Believe it or not there are legacy websites from decades ago still available on the internet.

You’ve likely used one many times.

Amazon.com is nearly the same site as it was years ago. It has been constantly updated over the years, and improved until today.

A complete redesign would have been too risky and disruptive for your bottom line.

It may be difficult for some legacy sites to create an XML sitemap. It’s no problem if they have a hard-coded HTML websitemap.

Google tells you to delete it. No.

Sites large

Mueller was correct when he said site navigation is essential. However, it is not always applicable to larger sites.

It is impossible to fit a complex website with thousands of pages into one menu.

Sitemaps that accurately reflect the site structure are helpful for humans and search engines to find sites more easily.

It’s not necessary to list each page. This would be too cumbersome.

Alternative to mega menus

Sites often try to squeeze as much information as possible into their site navigation, and end up with mega menus.

The navigation menu appears when you hover your mouse over it.

The sheer number of choices available to users can quickly overwhelm them visually.

Sometimes, users may lose their level one menu because they are not fast enough to point at the correct menu item.

Mega menus can be tempting, especially for large companies in which every stakeholder wants to be at the top.

They are, however, often a user-experience nightmare as Vitaly Friedman, founder of Smashing Magazine clearly illustrates.

Sites that do not allow internal search

You might find it difficult to reach people through web outreach.

Sites without an internal search function are often such a case. It is often difficult to find their contact page.

Although a search function would make sense, it is not a common solution.

Some theme owners or website owners remove the search input.

A visual sitemap is helpful if I cannot find a page standard in your menu or there is no search function.

Orphan pages

You will end up with many orphan pages if you can create a menu that is both usable and doesn’t take up too much screen space.

Orphan pages are pages that aren’t linked from within the content or the menu structure.

This is when content is removed or updated and links are lost.

Orphan pages are often best removed.

These links make sense, but they might not be as important to include in the main menu or footer.

These pages could be linked in a readable, visible sitemap to make it easy for people searching for them to find them.

Indexing is faster

Let’s face it, this is not fair! Google may not index a page or any other resource listed on an XML sitemap just because it is listed.

It is possible that your site does not have sufficient authority to be included in the Google index.

This was a very common problem for new sites in past years. Some content was not included in the index, especially since Google has less computing power and sites are newly created.

Although it is less common now, you may still want to link content within your organization to increase your chances of being indexed.

You can link to the most important content at the top of this page, prominently and/or more frequently.

Better link equity

When we speak of authority, we mean the link equity some SEOs still refer to as ” Link Juice“.

Link equity flows to some degree when there is an HTML sitemap showing some links at the top and some at the bottom.

Pages at the top of the page are expected to have more authority and be more authoritative than those at their bottom.

As far as I know, there is no hierarchy with XML sitemaps. It lists only the resources that are listed.

The top item is not the most important or of the greatest authority, but it’s the most recent.

Unless you program a hierarchy into XML sitemaps, there is no inherent hierarchy.

An HTML sitemap visible allows you to link the most important resources and send more authority and users their way. It is similar to your site navigation. It’s not the “latest” but it is the most important, ideal.

Useability

Sitemaps are important because they improve usability.

After all, this is Search Engine Land, not UX Collective. They recommend creating an internal UX sitemap to every website project, at the very least while you establish the information architecture.

HTML sitemaps enhance the user experience in particular:

• Accessibility.
• Discoverability.
• Findability.
• Usability.

These are the most important use cases for HTML websitemaps. They were the same as in the ’90s when Google began using links to index and rank and well before XML sitemaps were created.

Users older than me (I’m one of them) might prefer sitemaps with clear pages to navigate the website, rather than shaky menus or search tools that show irrelevant items.

Many elderly people are unable to use mouse-over or mouse-click effectively.

People with disabilities, especially those who are visually impaired, require as few mouse clicks as possible. A one-page overview may be the best option for them.

Examples of useful sitemaps in the wild

The New York Times

Because almost everyone knows the New York Times, it’s an authority.

I am going to give the NYT a great example because they did something right and stood out.

The NYT makes visible HTML sitemaps available to search users, readers, and search engine robots.

Many sitemaps have been highlighted by Chris Long , a fellow SEO from Go Fish Digital. This is an excellent example of sitemaps being used to benefit websites.

“60 Second SEO: The New York Times has the best HTML sitemaps that I have ever seen. This is an example of how a 1912 article can be found just 5 steps from their home page:

Long has done an outstanding job. I won’t go into too many details.

Apple.com

Apple.com is another great example of sitemap usage on a large website. The sitemap’s simple, plain text is easy to read and use by search engines and people.

Take note, however, of the differences between the sitemap and the actual navigation Apple has above the content.

The Apple sitemap begins with “About Apple” (not connected) and then the following links:

Site navigation begins with the shop and certain products in contrast.

Evidently, the SEO team made the shop and products visible to search engines and website visitors. However, the structure and importance of pages appears to be totally different.

The huge Apple sitemap provides a comprehensive overview of site contents and is not too overwhelming despite having more than 100 links.

They also have a footer that reflects the top menu structure, rather than the separate “sitemap” page.

Although it is smaller in terms of font size and the number listed links, it is still very useful.

No matter where you are, you will always have a sitemap or site navigation to assist you.

Apple’s website design is centered on a user experience that is “everything is easily found”.

Search Engine Land first published the post How and when to use HTML sitemaps in SEO and UX.