Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Why robots.txt if you want everything indexed?
buckworks - 2:50 pm on Mar 3, 2003 (gmt 0)
If you have a custom 404 page, having a robots.txt will conserve a bit of bandwidth.
Thread source:: http://www.webmasterworld.com/robots_txt/33.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com