Google Sitemap

Does it get crawled by non-google bots?

[soccrt]

Does the google sitemap get crawled by non-google bots, or does a webmaster need to create another one for other search engines?

[SEOtop10]

Generally you do not link to the Google sitemaps file (whether you name it sitemap.xml or anything else) on your site navigation. Therefore the other bots won't know about it.

If you did, they would know about this file and spider it. However as of date, no bot can interprete this file and crawl the included URLs. Even Google bot does not.

You have to submit your file to the Google Sitemaps page and a special software interpretes this file and prepares a list of files to crawl, that is given to the Gbot.

[soccrt]

I have done this and it has started to work a little. How can I create a sitemap for other bots to crawl?

[soccrt]

How can I create a sitemap for other bots to crawl as google only makes up for 35% of the SE market.

[webdude]

That's an interesting question. If you read some of the documentation, it is worded in such a way to leave you believing that the sitemap affects more then just Google.

Case in point. This is stated on the protocol pages. You might even find more ambiguities...

Please note that the Sitemap Protocol supplements, but does not replace, the crawl-based mechanisms that search engines already use to discover URLs. By submitting a Sitemap (or Sitemaps) to a search engine, you will help that engine's crawlers to do a better job of crawling your site.

mmm... Sounds like they are referring to more search engines then just Google.

And then this...

The Sitemap Protocol allows you to inform search engines about URLs on your websites that are available for crawling. In its simplest form, a Sitemap that uses the Google Sitemap Protocol is an XML file that lists URLs for a site.

Once again, it sound like they are talking of multiple search engines.

And this, on the FAQ page...

Your Sitemap should be placed in a location on your web server that is accessible to our crawlers. If you'd like to prevent other robots from accessing your Sitemap file, you may use a robots.txt file that complies with the Robots Exclusion Standard.

Other robots? Are they implying that other bots can crawl this page?

Good question!

[SebastianX]

>Does it get crawled by non-google bots?
No. It gets fetched by ia_archiver after browser views with the Alexa toolbar installed, and probably here and there by other bots by mistake (following a link). There is not yet any engine making use of it for crawling. The sitemap protocol is open for all engines, and Google hopes others will make use of it. After Yahoo's Site Explorer announcement probably Y! will not implement it (soon), and it's not clear what the other engines think about it. A bit early for speculations, it's running 2 months in BETA now.