Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Why should I have a robots.txt file?
DamonHD - 5:04 pm on Dec 30, 2006 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/3203372.htm
My feeling is, in the spirit of KISS (Keep It Simple, Stupid), that unless you NEED it, don't put it up at all, and filter out any (spurious) errors from its absense in other ways.
Brought to you by WebmasterWorld: http://www.webmasterworld.com