Page is a not externally linkable
Bewenched - 3:19 pm on Apr 12, 2007 (gmt 0)
Yes I know that they ignore the sitemap, but putting this in our robots.txt will give them a complete roadmap to follow.
Site scrapers already ignore robots.txt, so no need to worry. Just focus on the search engines