Tapolyai - 8:46 pm on Nov 17, 2006 (gmt 0)

I believe this has been implemented both in Google and Yahoo! at least since late '05. I have been using the site maps since November of 2005.

I think this will not just reduce the spider technical overhead, but potentially create a legal protection for the search engines.

If the spiders can be targeted by the webmasters' direction through sitemaps (verses 'just' scanning through everything), the webmasters no longer have a case to claims of 'illegal copying' or similar.

[edited by: Tapolyai at 9:01 pm (utc) on Nov. 17, 2006]