Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Why should I have a robots.txt file?
netmeg - 6:45 pm on Dec 31, 2006 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/3203372.htm
Some of my sites get hit really hard by spiders (specially overseas) that have no reason to be spidering them - sucking up a ton of bandwidth - and we block for that reason.
Brought to you by WebmasterWorld: http://www.webmasterworld.com