sitemap.xml - does it help, and are there risks?

[con771]

would it help googles crawling and indexing if I added a sitemap.xml file on my site?
Any risks?

[tedster]

Some people report a faster spidering of new urls with a sitemap.xml file, especially for large sites that add new urls frequently. However, there is no guaranteee that spidered=indexed, Google still makes their own decision on what urls to actually add. Even spidering frequency is still determined by the crawl team's own math - it's just that the xml sitemap seems to get priority over a natrual crawl in some cases.

As far as downsides, most of the troubles I've heard of come from technically inaccurate sitemap generators. You want to be sure that its not including diferent urls that resolve to the same content. This has particularly been a challenge for some sites that are using a rewrite scheme and only want to see the "friendly" urls in the index - but the generator also picks up the "unfriendly" urls that use a query string.

[piatkow]

I created a site map at the same time as a couple of other enhancements to the site. The overall effect was good but I don't know how much was down to the site map.

[punisa]

Hi,
Google does encourage using the sitemap, I would say use it. I think it does have the effect on the crawl speed.
Another useful thing is that by using a sitemap you have the knowledge how many articles/pages you have. and then you can run site:yoursite.com in google to check how much (in number) of those has been indexed.

One thing that I need to mention is that query string URL's should never be available to anyone if you are using mod rewrite ! If a crawler tool can find so can google !

[edited by: tedster at 6:00 pm (utc) on May 22, 2008]

[gogame]

i don't know why,my sitemap own more than 1500 urls but only 28 urls included from google webmastertools,then i check it via site:mydomain.com,it shows 1,100 urls indexed,what's the difference?why do they get that great gap?

[tedster]

Hello gogame, and welcome to the forums.

I'd trust the site: operator results over Webmaster Tools, which unfortunately tends to give buggy numbers more frequently. After all, site: gives actual proof that the url is indexed, whereas Webmaster Tools is a secondary reporting utility. More steps involved means more potential points of failure.

Also, it's very unlikely that Google will actually index every url in sitemap.xml file. Spider them all? Maybe, but spidering is no guarantee that a url will stick in the index. However, you should be able to hit 80% - 90% indexed with a strong website, and it sounds like you're in that area.

[edited by: tedster at 4:04 am (utc) on May 26, 2008]

[gogame]

Thanks Tedster,your saying relaxs me,also i'll try to get more sitemap urls indexed.

Glad to see you at the eight-year's time today since your join on May 26,2000,thanks your great contribution to webmasterworld,anyway.

[zaqwsx3]

However, you should be able to hit 80% - 90% indexed with a strong website

Would love to know how many people out there have index rates around this mark? (I'm well short of it but will set it as a goal)

The only comment from Google I have found is from the Help Center where they state, "While we can't guarantee that all pages of a site will consistently appear in our index, we do offer our guidelines for maintaining a Google-friendly site".