- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Add Sitemap To robots.txt For Autodiscovery
Bewenched - 3:19 pm on Apr 12, 2007 (gmt 0)
|Site scrapers already ignore robots.txt, so no need to worry. Just focus on the search engines |
Yes I know that they ignore the sitemap, but putting this in our robots.txt will give them a complete roadmap to follow.
Thread source: http://www.webmasterworld.com/robots_txt/3310622.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com