Welcome to WebmasterWorld Guest from 35.175.180.108

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

HTML Sitemap - Do we still need it?

     
8:01 am on Feb 20, 2016 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 8, 2003
posts: 516
votes: 4


Is a sitemap (html) still relevant? Dont have one for my website, thinking of making one which would have categories, monthly archives and tags. Not many sites have them now, but then again I do think apart from the XML file there has to be a single page map which directs the bot to all the pages. Your thoughts?
7:50 pm on Feb 20, 2016 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member andy_langton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 27, 2003
posts:3332
votes: 140


A sitemap has never really been a "requirement" unless a site did not have an effective link structure. The requirement is that every page have a least one link to it from another page, and an HTML sitemap is one way to 'guarantee' that. However, decent navigation combined with contextual linking will always work better because it provides the opportunity to link to more important pages more frequently and more prominently. A sitemap is, by necessity, a "flat" series of links.

That said, your users might find a sitemap helpful in getting an overview of your entire site, or as a fall-back if they can't find what they're looking for. How useful this is will depend on your site and how you make the sitemap.
7:52 pm on Feb 20, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15932
votes: 887


Are you asking about humans or search engines?

A sitemap is needed for search engines if you've got content that they might not otherwise find, such as pages at the far end of a complicated script-based link.

A sitemap is needed for humans if you've got ... oh, wait, I said that already. And then, of course, you have to make sure your scriptless humans can easily find the sitemap.

a single page map which directs the bot to all the pages
I really hope you mean, specifically, search engines, because why on earth would you want to be helpful to the average robot on the street?
9:56 am on Feb 21, 2016 (gmt 0)

New User

joined:Feb 4, 2016
posts:12
votes: 0


it depends how large your website is..If this is an ecommerce site, then I would recommend to create one..

If this is just 5 or 10 pages static site, it's fine not to create sitmap..
10:20 am on Feb 21, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10560
votes: 1122


I've never put one up, then again, I make sure my internal linking makes sense. Keep it simple.
5:59 am on Feb 22, 2016 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 8, 2003
posts: 516
votes: 4


its for the search engine. I have a Wordpress blog, its fairly large and has close to 12,000 entries. So was contemplating on a sitemap for the older entries.
6:38 am on Feb 22, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10560
votes: 1122


Do you have any evidence the se desired does not have these files indexed? Do a site:example.com and see what you get.

These days, and almost always, trying to fix something that ain't broke will only cause trouble.
6:34 pm on Feb 23, 2016 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:July 29, 2007
posts:2014
votes: 215


I have one on a smaller site, 300 pages or so, and it seems to help visitors, it gets used a bit. I have the contact information tucked away inside it and so don't have a contact link sitewide.
5:05 am on Feb 24, 2016 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 8, 2003
posts: 516
votes: 4


@tangor I am guessing all the posts are indexed from the xml sitemap. But then I am guessing the google bot reaching the page by minimal hopping from the website also makes a difference in the ranking of the said content.
2:30 am on Mar 15, 2016 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:July 29, 2007
posts:2014
votes: 215


Search bots visit URLs directly, they don't click around. There is a specific bot that does appear to click around and even interacts with forms but it's not there to rank the site, only to test it for nefarious intent.

Navigation for people is over-rated, we're moving to an endless scroll system being the preferred way to find content. I'd expect voice to push that even further, soon.
7:28 am on Mar 15, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10560
votes: 1122


Links are links. If you have internal links to your content visible in group or cluster, any se worth its salt will find your stuff. What the heck is a sitemap? Your internal links ... if you've done your job properly. CMS sites do this (most of the time). Static sites take a bit more work (to remember your link structure). Run a link checker to make sure you don't have orphaned pages (ie, do not link back to home) which can be found by external linkage (others link to you) and don't report "home" (your site). I've done that a few times in the past. (arrgh!)

The day of a sitemap has passed. these days I suspect that all SEs ignore them as nothing more than a suggestion. Their tools will do the grunt work of finding everything, not just the stuff you want them to find.
11:50 am on Mar 16, 2016 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 21, 2004
posts: 67
votes: 0


I've always worked under the assumption that an HTML sitemap passes PR, while the XML sitemap is for indexing. Thoughts?
2:27 pm on Mar 16, 2016 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4558
votes: 363


I view the html sitemap as useful to visitors looking for some specific information that is not commonly in your navigation. I use it for a more user friendly navigation for visitors for the whole site. If someone wants to find your shipping policies for example, or product care or FAQ pages that won't usually be in the navigation or indexed.

The xml sitemap for bots won't have those pages either. Google says a sitemap is to list the pages you want indexed and as far as I can tell the Bingbot uses sitemaps (and robots.txt) to feed their robots.
10:49 pm on Mar 16, 2016 (gmt 0)

Junior Member

Top Contributors Of The Month

joined:Oct 17, 2015
posts: 111
votes: 32


I requested G put an option in the sitemap section in WMT for "obviously we get things wrong, if you'd prefer we stopped poking around in every corner of you site, indexing rubbish you banned us from or didn't even know existed, tick this box to ensure we stick to sitemap entries only and we'll stop causing you an headache and hours of removal time and worry with regards to ranking because of pages and other tech rubbish G indexed incorrectly."

A sitemap in any form is required so that update frequency is known and robots see all pages.
11:47 pm on Mar 16, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10560
votes: 1122


The search engine's job is to find everything all by itself. They don't need sitemaps ... which is why a few webmasters are dismayed that some of their stuff has been found. :)
12:21 am on Mar 17, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15932
votes: 887


tick this box to ensure we stick to sitemap entries only

Sitemaps are not exclusive ("index ONLY this stuff"). They are inclusive ("be sure not to overlook this stuff"). Search engines don't even have to look at them if they don't feel like it. It's the webmaster's responsibility to apply noindex tags where needed.
2:53 am on Mar 17, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10560
votes: 1122


lucy24 is spot on, kiddies! So ... this does beg the question: are sitemaps still valid, or even desired?

Clients that ask for them I do it. Client with no preference I don't. Looking back over the years while serving both I do not see any statistical plus or minus either way.
12:36 pm on Mar 17, 2016 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 8, 2003
posts: 516
votes: 4


Not sure but there would be a search engine metric which would be valuing the importance of a page based on its internal linking.

My website has close to 19,000 articles so was a tad worried for the older ones.
7:48 pm on Mar 17, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


I keep an HTML site map page linked from the bottom of every other page. This is for visitors however it also ensures the SEs can find everything. This is also the page I always "fetch as Googlebot/Bingbot" and often check the box to "URL and linked pages submitted to index."
9:12 pm on Mar 17, 2016 (gmt 0)

Junior Member

Top Contributors Of The Month

joined:Oct 17, 2015
posts: 111
votes: 32


tangor and lucy, you can put noindex. G doesn't have to follow your rule and doesn't. Hence why I wrote what I did. kiddies... Running a website and doing things properly, only to get screwed by google because they include content you didn't know existed, what's wrong with a stick to sitemap command? noindex would be fine if g stuck to it but then it goes off looking for something else to index in replace of that.

And when it comes to removal. It then removes ten pages every other day and takes months.
9:38 pm on Mar 17, 2016 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2000
posts:12401
votes: 410


you can put noindex. G doesn't have to follow your rule and doesn't.
timemachined... are you by any chance using noindex and robots.txt together? This can lead to problems, as they in a sense 'conflict'.

Meta robots "noindex" is the recommended way to keep a url out of Google's displayed index... and robots.txt is the recommended way to keep a url from being spidered. However, if you use them both together, Googlebot will not spider the url and thus will not see "noindex". Often, in such cases, the url can be displayed as a reference, which is just what you don't want to have happen. It's a sometimes difficult point.

Here's a long but thorough discussion on the topic, which, if "noindex" not working is the problem you're having, I highly recommend...

Pages are indexed even after blocking in robots.txt
https://www.webmasterworld.com/google/4490125.htm [webmasterworld.com]