homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Should I noindex my site map/s?
I want robots to follow links, but not index the actual site map.

 10:37 am on Jun 20, 2009 (gmt 0)

I am setting up about 5-8 HTML pages to be used as a site map for a Q&A section of my site. A few of the pages do have a little bit of original content on them, but mainly it is a glorified site map.

Although the pages will be linked to from my main web site, I don't really see a huge advantage in my visitors arriving at these pages via a search engine and so I was thinking of adding the <meta name="robots" content="noindex"> tag to them.

I'm assuming that Google and other SEs will still follow links on these pages? But will they index those pages properly? Or will they see them as devalued somehow?

Any thoughts on this, or other suggestions?



 11:02 am on Jun 20, 2009 (gmt 0)

Site map means one thing. Pages with links means another thing. But noindex is something entirely different... and should be your answer.

Conversely, I wouldn't create pages with links as a site map unless I intended them to be used that way by humans or bots, and I would want them indexed.

Different strokes for different folks.


 1:58 pm on Jun 20, 2009 (gmt 0)

Thanks tangor. Although, I don't really understand your post.

As I said, I intend the pages with links to be used by both humans and bots to find information about a section of my site (call them site maps, or not, the function is the same - albeit, it is for a sub-section of the site).

The issue I am posting about relates to whether I want the pages to be an entry point to the site from search engines. I have linked to them from the intro page on the Q+A section of my site as a reference, and they would make sense to visitors if viewed via that route. They would make less sense if the first thing a visitor saw was a list of links.

So my suggested solution is to exclude them from SE indexing (but not from following links), which I hope will serve both purposes. My concern is whether there might be any downsides to this.


 7:57 pm on Jun 24, 2009 (gmt 0)

There are two different artifacts that get called site maps. Regardless of technical meaning that is common usage.

1. An XML site map which is intended purely for the use of search engines. I think this is what Tangor is referring to.
2. An HTML site map which is effectively a table of contents of your site intended primarily for human use but which will incidentally be spidered by search engines.

The noindex attribute may be ignored by some search engines but should be good for Google. I am not aware of any issues. It really is a matter of personal choice.


 9:31 am on Jun 26, 2009 (gmt 0)

Thanks piatkow.

Still interested to know if anyone has any thoughts as to whether links from a noindex page might carry less weight somehow.


 12:13 pm on Jun 26, 2009 (gmt 0)

I wonder if the value of their anchor text is dimished, as technically, the text isn't indexed - or is it, but just not shown in public?


 4:59 am on Jun 28, 2009 (gmt 0)

Indeed. In theory the text is not indexed, but the links are followed. In practice, to follow a link, a spider would have to at least 'read' the page to know that a link exists. Then, presumably, the spider 'forgets' the page text. But does it also forget the link text?

Or, is the interpretation simply that the page is 'read' and 'indexed', not simply shown in the public index. Either way, it's hard to know how this affects the 'value' of the link.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved