HTML Sitemap - Do we still need it?

[morpheus83]

Is a sitemap (html) still relevant? Dont have one for my website, thinking of making one which would have categories, monthly archives and tags. Not many sites have them now, but then again I do think apart from the XML file there has to be a single page map which directs the bot to all the pages. Your thoughts?

[Andy Langton]

A sitemap has never really been a "requirement" unless a site did not have an effective link structure. The requirement is that every page have a least one link to it from another page, and an HTML sitemap is one way to 'guarantee' that. However, decent navigation combined with contextual linking will always work better because it provides the opportunity to link to more important pages more frequently and more prominently. A sitemap is, by necessity, a "flat" series of links.

That said, your users might find a sitemap helpful in getting an overview of your entire site, or as a fall-back if they can't find what they're looking for. How useful this is will depend on your site and how you make the sitemap.

[lucy24]

Are you asking about humans or search engines?

A sitemap is needed for search engines if you've got content that they might not otherwise find, such as pages at the far end of a complicated script-based link.

A sitemap is needed for humans if you've got ... oh, wait, I said that already. And then, of course, you have to make sure your scriptless humans can easily find the sitemap.

a single page map which directs the bot to all the pages

I really hope you mean, specifically, search engines, because why on earth would you want to be helpful to the average robot on the street?

[anneconq]

it depends how large your website is..If this is an ecommerce site, then I would recommend to create one..

If this is just 5 or 10 pages static site, it's fine not to create sitmap..

[tangor]

I've never put one up, then again, I make sure my internal linking makes sense. Keep it simple.

[morpheus83]

its for the search engine. I have a Wordpress blog, its fairly large and has close to 12,000 entries. So was contemplating on a sitemap for the older entries.

[tangor]

Do you have any evidence the se desired does not have these files indexed? Do a site:example.com and see what you get.

These days, and almost always, trying to fix something that ain't broke will only cause trouble.

[JS_Harris]

I have one on a smaller site, 300 pages or so, and it seems to help visitors, it gets used a bit. I have the contact information tucked away inside it and so don't have a contact link sitewide.

[morpheus83]

@tangor I am guessing all the posts are indexed from the xml sitemap. But then I am guessing the google bot reaching the page by minimal hopping from the website also makes a difference in the ranking of the said content.

[JS_Harris]

Search bots visit URLs directly, they don't click around. There is a specific bot that does appear to click around and even interacts with forms but it's not there to rank the site, only to test it for nefarious intent.

Navigation for people is over-rated, we're moving to an endless scroll system being the preferred way to find content. I'd expect voice to push that even further, soon.

[tangor]

Links are links. If you have internal links to your content visible in group or cluster, any se worth its salt will find your stuff. What the heck is a sitemap? Your internal links ... if you've done your job properly. CMS sites do this (most of the time). Static sites take a bit more work (to remember your link structure). Run a link checker to make sure you don't have orphaned pages (ie, do not link back to home) which can be found by external linkage (others link to you) and don't report "home" (your site). I've done that a few times in the past. (arrgh!)

The day of a sitemap has passed. these days I suspect that all SEs ignore them as nothing more than a suggestion. Their tools will do the grunt work of finding everything, not just the stuff you want them to find.

[spiral]

I've always worked under the assumption that an HTML sitemap passes PR, while the XML sitemap is for indexing. Thoughts?

[not2easy]

I view the html sitemap as useful to visitors looking for some specific information that is not commonly in your navigation. I use it for a more user friendly navigation for visitors for the whole site. If someone wants to find your shipping policies for example, or product care or FAQ pages that won't usually be in the navigation or indexed.

The xml sitemap for bots won't have those pages either. Google says a sitemap is to list the pages you want indexed and as far as I can tell the Bingbot uses sitemaps (and robots.txt) to feed their robots.

[timemachined]

I requested G put an option in the sitemap section in WMT for "obviously we get things wrong, if you'd prefer we stopped poking around in every corner of you site, indexing rubbish you banned us from or didn't even know existed, tick this box to ensure we stick to sitemap entries only and we'll stop causing you an headache and hours of removal time and worry with regards to ranking because of pages and other tech rubbish G indexed incorrectly."

A sitemap in any form is required so that update frequency is known and robots see all pages.

[tangor]

The search engine's job is to find everything all by itself. They don't need sitemaps ... which is why a few webmasters are dismayed that some of their stuff has been found. :)

[lucy24]

tick this box to ensure we stick to sitemap entries only

Sitemaps are not exclusive ("index ONLY this stuff"). They are inclusive ("be sure not to overlook this stuff"). Search engines don't even have to look at them if they don't feel like it. It's the webmaster's responsibility to apply noindex tags where needed.

[tangor]

lucy24 is spot on, kiddies! So ... this does beg the question: are sitemaps still valid, or even desired?

Clients that ask for them I do it. Client with no preference I don't. Looking back over the years while serving both I do not see any statistical plus or minus either way.

[morpheus83]

Not sure but there would be a search engine metric which would be valuing the importance of a page based on its internal linking.

My website has close to 19,000 articles so was a tad worried for the older ones.

[keyplyr]

I keep an HTML site map page linked from the bottom of every other page. This is for visitors however it also ensures the SEs can find everything. This is also the page I always "fetch as Googlebot/Bingbot" and often check the box to "URL and linked pages submitted to index."

[timemachined]

tangor and lucy, you can put noindex. G doesn't have to follow your rule and doesn't. Hence why I wrote what I did. kiddies... Running a website and doing things properly, only to get screwed by google because they include content you didn't know existed, what's wrong with a stick to sitemap command? noindex would be fine if g stuck to it but then it goes off looking for something else to index in replace of that.

And when it comes to removal. It then removes ten pages every other day and takes months.

[Robert Charlton]

you can put noindex. G doesn't have to follow your rule and doesn't.

timemachined... are you by any chance using noindex and robots.txt together? This can lead to problems, as they in a sense 'conflict'.

Meta robots "noindex" is the recommended way to keep a url out of Google's displayed index... and robots.txt is the recommended way to keep a url from being spidered. However, if you use them both together, Googlebot will not spider the url and thus will not see "noindex". Often, in such cases, the url can be displayed as a reference, which is just what you don't want to have happen. It's a sometimes difficult point.

Here's a long but thorough discussion on the topic, which, if "noindex" not working is the problem you're having, I highly recommend...

Pages are indexed even after blocking in robots.txt
https://www.webmasterworld.com/google/4490125.htm [webmasterworld.com]