Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Add Sitemap To robots.txt For Autodiscovery


blend27 - 3:25 pm on Apr 12, 2007 (gmt 0)


Bewenched, the sitemap.xml could cloacked against IP Ranges. The normal user should not make requests nor looking for it to your sitemap.xml file.
As long as SE's don't provide CASHED Copy it's a cool feature.

Otherwise it is as you say: heaven for scrapers,

but then again: too many requests(2 is toooo many) with no JS and NO Images: 1 warnning, then Block and wait till the next episode....


Thread source:: http://www.webmasterworld.com/robots_txt/3310622.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com