homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 66 message thread spans 3 pages: 66 ( [1] 2 3 > >     
site maps - does Googlebot recognize them as such?
What is the best url/link text for a site map?

 2:28 am on Oct 30, 2003 (gmt 0)

Does Googlebot recognize a site map as such or does it see it merely as another page with a lot of links on it? Do the words "site map" or "sitemap" in the link text or the url of a site map page mean anything to Googlebot? Is there something specific the bot looks for?

The reason for the question is I have three site maps with urls crawler1.php, crawler2.php, and crawler3.php, with the words "site map" included in the link texts from the index page. These are only infrequently visited by Google. Is there something specific I should change them to?

My apologies if this has been covered before, but a search did not turn up anything on this question.




 7:35 am on Oct 30, 2003 (gmt 0)

I like to avoid anything generic when it comes to techniques that could potentially be open to abuse.

I'm not saying that Google ignores or penalizes or doesn't spider site maps, I just don't risk it.

My suggestion would be for you to create another clever name for "site map", such as "site structure", "keyword pages", "keyword structure", etc.

This way you would definitely be under the radar. Just try to make sure you don't have too many URL's per page.


 9:32 am on Oct 30, 2003 (gmt 0)

I was under the impression that site maps were a very normal (if not encouraged) part of web design.

I can hardly see how using a site map could be open to any sort of major abuse.


 9:45 am on Oct 30, 2003 (gmt 0)

Surley if it good enough for Google it is good enough for web designers and site owners!



 10:20 am on Oct 30, 2003 (gmt 0)

You gotta cringe when the site map ranks higher than specific pages ;)



 11:43 am on Oct 30, 2003 (gmt 0)

I was under the impression that site maps were a very normal (if not encouraged) part of web design.

Google [google.com] definitely encourages their use:

Offer a site map to your users with links that point to the important parts of your site. If the site map is larger than 100 or so links, you may want to break the site map into separate pages.

I see absolutely no reason for hiding the fact that a page is indeed a site map. It is useful both for spiders and for human users.


 11:46 am on Oct 30, 2003 (gmt 0)

You could also check out the sitemaps of large websites which are doing well in Google and learn some tips.


 3:45 pm on Oct 30, 2003 (gmt 0)

G loves sitemaps, and they can act as another legitimate front door to a website, though that was something in my distant past.

Nowadays, i use them to give a structural view of the underpinnings of the site, with the structure being embedded in a straight forward manner.


 3:46 pm on Oct 30, 2003 (gmt 0)

I changed the name recently to site_map.htm and have had much better results with the spiders google in particular.


 3:58 pm on Oct 30, 2003 (gmt 0)

A sitemap will definetly help you also I should rename all yoursite maps t site_map.asp and sitemap.asp and so far.

good luck with it


 4:12 pm on Oct 30, 2003 (gmt 0)

"I'm not saying that Google ignores or penalizes or doesn't spider site maps, I just don't risk it."

Google actually recommends the use of site maps.


 4:31 pm on Oct 30, 2003 (gmt 0)

Indeed Google does appear to encourage site maps. However, at the risk of opening a can of worms...

The advice from 2_much could be a good one to adhere to. Using less than 100 links per page seems to be the past consensus, however I could in no way say that is a firm cut-off point or not.

The Google toolbar is playing funny games at the moment, however I would say with some degree of certainty that some site_maps I can see at the moment have been applied a PR downgrade or are not listed at all.

That said, if you are making a site map for the right reasons, that is for the benefit of your users, then maybe it's not worth the worry.


 5:24 pm on Oct 30, 2003 (gmt 0)

Site Maps are for users to navigate in your site..so there is no way that google will ban those pages...

but you have to make sure that whatever you do to the sitemap is legit in terms of google.

Anything spamy from any page that includes sitemap page can get penalized or banned...


 5:37 pm on Oct 30, 2003 (gmt 0)

I have not been able to detect a difference in spidering behavior between sitemaps and other pages. Their purpose for Googlebot is only to find your other pages.

Spidering frequency is in line with the rest of the site.


 6:42 pm on Oct 30, 2003 (gmt 0)

I see sitemaps as a key part of the whole 'spider feeding' strategy. By their very nature, sitemaps should have the largest / or atleast joint largest amount of internal inbound links on a site, with the exception of the front page, if linked from every page.

Make sure though that each page is less than 100K and contains less than 100 links -- I usually go for less than 80 links.

I strongly recommend their use, as does Google and every other SE I know of, and see no reason why you should try to hide their existance, indeed they should be promoted!


 7:05 pm on Oct 30, 2003 (gmt 0)

I'm not saying that Google ignores or penalizes or doesn't spider site maps, I just don't risk it.

I agree, why risk it? I design mine to look just like a DMOZ category. Anchor text & a 1 or 2 sentence description for each link. Keeps the sitemap from looking like a links page & helps your users understand your anchor text.


 7:07 pm on Oct 30, 2003 (gmt 0)

Risk what?

My site map is named "site_map.html", weighs in at 16kb, contains 105 full paths to my main pages using anchor text links, and has a PR5. I SSI this to the bottom of every page on the domain, and when I've completed a significant site update, I submit site_map.html to the Google add URL utility which usually results in a full crawl.

Yes, I'd say Google supports site maps, says so here [google.com].


 7:14 pm on Oct 30, 2003 (gmt 0)

Risk what?

The page not ranking as well as it possibly can.


 7:17 pm on Oct 30, 2003 (gmt 0)

Site maps are not only not a risk it is bad SEO and bad developing to not have one. It's about as silly as saying you don't need a domain all you need is an ip.


 7:52 pm on Oct 30, 2003 (gmt 0)

Site maps are considered a best practice in many circles. My understanding is that Google is a great fan of best practices.

Unfortunately, I think they just treat site maps like regular pages...which is why they suggest limiting the page to 100 links. Personally, I think that, if a site is large, then it should have a large site map. I hate the collapsable site maps designers use to get around the Google 100 link limit.

It would be great to have a meta tag that says: "This is a site map or site index" ... and for Google to accept that the site map can be one page with a thousand internal links.

As my site map [slsites.com] has well over 200 links, I've been wondering if I should hide the site map from Google. When given a choice between designing for users or designing for Google, I usually err on the side of the user...which is probably why I get a good amount of repeat traffic and not that much Google traffic.


 7:56 pm on Oct 30, 2003 (gmt 0)

No external links please. It's against the TOS.


 8:50 pm on Oct 30, 2003 (gmt 0)

Risk what?
The page not ranking as well as it possibly can.

skipfactor, did you read this [google.com]?


 9:44 pm on Oct 30, 2003 (gmt 0)

On my own personal website, i have a site map that was created just to help insure the bots have one more way to find all my pages. My site map is note even visible to users, don't really need one.
I named my site map something like "spiderfood.htm", it's just my sense of humor, i need it. The site map page has a noindex,follow tag. It has not been listed but is repeatedly visited by the bots. Almost all of the pages have been indexed in google and the site is only a couple months old.

Im not recommending you name your site map "spiderfood", and i'm not even saying you should make one just for the se's. I'm only making a point.


 10:46 pm on Oct 30, 2003 (gmt 0)

Another very good reason for a site map is for spreading PR around. Because you link from every page back to sitemaps, sites maps pages get same PR as index page, therefore each link on the site map gets PR-1.

However, if you have more than say 70-100 links on that page, the PR attributed may be PR-2. Therefore, a good reason to keep number of links down.

Site maps pages are also fab for getting the power of text links working. For a page that is ranking low on a particular phrase, I always add a few more links with those phrases to bump it up.

I read some good advice - see where onpage optimisation will get you for a phrase, then increase the specific text links until you get to the top.


 12:00 am on Oct 31, 2003 (gmt 0)

"Because you link from every page back to sitemaps, sites maps pages get same PR as index page"

This should very rarely be the case; only when an index page has a high PR5 (for example) and the site map a low PR5.

Index pages pretty much invariably have external links while site maps pretty much never do, so index pages will always have considerably higher PR. Not that this is a bad thing or anything, site maps are great ideas, and good ways to ensure a minimum level of PR for all your pages.


 12:12 am on Oct 31, 2003 (gmt 0)

Many thanks for all the comments. I haven't replied earlier because I live in a different time zone to most.

To clear up one point: I was not trying to hide the site maps, far from it. I want to make them stand out and attractive as possible for Googlebot to grab them. They are all under 100 links per page, under 22K, structured to be helpful to users, and they also follow Googles own site map format with brief headings followed by the appropriate links. Unfortunately Google rarely visits them.

As for naming convention, I'm still confused. Some people are reporting good take up with "site_map", whereas others seem to be saying that Google treats site maps like a normal page whatever they are called. And of course Google uses "sitemap".

I am actively considering renaming them, but it seemed sensible to canvas for opinions on the most Google-attracting name before proceeding.

Presumably if "site_map" is effective, so would be "site-map" - I hate underscores. :)



 12:24 am on Oct 31, 2003 (gmt 0)

don't use "site map". Give it an appropriate and descriptive but short name.


 12:33 am on Oct 31, 2003 (gmt 0)

A page is a page is a page. Having a different title means nothing, its how many clicks from the index page, or pages that have external links that is the more interesting point. Its the PR of those pages that also has an effect on how many times Google looks at it. Been said many times that Google looks at pages, not sites, even less so does it differentiate between title names.

re PR of pages - when I was PR5 on index, all 5 pages on menu structure were PR5 (200 page site). When index went to PR6, after a month the menu structure pages went to PR6. This last week subpages dipped to PR5 for about 3 days, now back to PR6. Large number of links have been added in last month, both onto index page and subpages. I keep number of menu pages small so more chance of keeping the max, and put javascript links on non essential pages, ie terms & conditions, contact us, ... I have another site with 50 links on every page via css/li menu structure. Since so many links on every page, PR5 index, PR4 on every internal but the value of 50 text links on every page of a 50 page site means it is top for almost every term without the need for extensive external linking to internal pages.

[edited by: t2dman at 12:45 am (utc) on Oct. 31, 2003]


 12:33 am on Oct 31, 2003 (gmt 0)

don't use "site map". Give it an appropriate and descriptive but short name.

For a 'map' of your website how more appropriate and short can you get?

The widely recognized term is sitemap, no reason to try something else.


 12:45 am on Oct 31, 2003 (gmt 0)

don't use "site map". Give it an appropriate and descriptive but short name.

I'm puzzled. What could be more appropriate and descriptive than "site-map" or "site_map"? The link text can be "site map" (unhyphenated) which is the recognized term found in numerous sites.

As I have more than one site map, in pracice they would have to be called soemthing like site-map-1, site-map-2, etc, or possibly more specifically site-map-red-widgets, site-map-green-widgets, etc.


This 66 message thread spans 3 pages: 66 ( [1] 2 3 > >
Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved